System and method for populating a virtual shopping cart based on a verification of algorithmic determinations of items selected during a shopping session in a physical store

ABSTRACT

An apparatus includes a display and a processor. The processor displays a virtual shopping cart. The processor also receives information indicating that an algorithm determined that a physical item was selected by a person during a shopping session in a physical store, based on a set of inputs received from sensors located within the store. In response, the processor displays a virtual item, which includes a graphical representation of the physical item. The processor additionally displays a rack video captured during the shopping session by a rack camera located in the store. The rack camera is directed at a physical rack located in the store, which includes the physical item. In response to displaying the rack video, the processor receives information identifying the virtual item, where the rack video depicts that the person selected the physical item. The processor then stores the virtual item in the virtual shopping cart.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of:

U.S. patent application Ser. No. 16/664,470 filed Oct. 25, 2019, byMatthew Raymond Magee et al., and entitled “CUSTOMER-BASED VIDEO FEED”;

U.S. patent application Ser. No. 16/664,490 filed Oct. 25, 2019, byShahmeer Ali Mirza et al., and entitled “SYSTEM AND METHOD FORPRESENTING A VIRTUAL STORE SHELF THAT EMULATES A PHYSICAL STORE SHELF”;

U.S. patent application Ser. No. 16/938,676 filed Jul. 24, 2020, byShahmeer Ali Mirza et al., and entitled “FEEDBACK AND TRAINING FOR AMACHINE LEARNING ALGORITHM CONFIGURED TO DETERMINE CUSTOMER PURCHASESDURING A SHOPPING SESSION AT A PHYSICAL STORE”, which is a continuationof U.S. patent application Ser. No. 16/794,083 filed Feb. 18, 2020, byShahmeer Ali Mirza et al., and entitled “FEEDBACK AND TRAINING FOR AMACHINE LEARNING ALGORITHM CONFIGURED TO DETERMINE CUSTOMER PURCHASESDURING A SHOPPING SESSION AT A PHYSICAL STORE”, now U.S. Pat. No.10,810,428 issued Oct. 20, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/663,564 filed Oct. 25, 2019, by Shahmeer AliMirza et al., and entitled “FEEDBACK AND TRAINING FOR A MACHINE LEARNINGALGORITHM CONFIGURED TO DETERMINE CUSTOMER PURCHASES DURING A SHOPPINGSESSION AT A PHYSICAL STORE”, now U.S. Pat. No. 10,607,080 issued Mar.31, 2020;

U.S. patent application Ser. No. 17/021,011 filed Sep. 15, 2020, byMatthew Raymond Magee et al., entitled “SYSTEM AND METHOD FOR POPULATINGA VIRTUAL SHOPPING CART BASED ON VIDEO OF A CUSTOMERS SHOPPING SESSIONAT A PHYSICAL STORE”, which is a continuation of U.S. patent applicationSer. No. 16/663,589 filed Oct. 25, 2019, by Matthew Raymond Magee etal., entitled “SYSTEM AND METHOD FOR POPULATING A VIRTUAL SHOPPING CARTBASED ON VIDEO OF A CUSTOMER'S SHOPPING SESSION AT A PHYSICAL STORE”;and

U.S. patent application Ser. No. 16/664,529 filed Oct. 25, 2019, byMatthew Raymond Magee et al., entitled “TOOL FOR GENERATING A VIRTUALSTORE THAT EMULATES A PHYSICAL STORE”, which are all incorporated hereinby reference.

TECHNICAL FIELD

This disclosure relates generally to remote monitoring techniques, andmore particularly, to a system and method for populating a virtualshopping cart based on a verification of algorithmic determinations ofitems selected during a shopping session in a physical store.

BACKGROUND

During a traditional shopping session in a physical store, a customerselects items from shelves located within the store and then presentsthose items to a cashier. The cashier generates a bill for the items andreceives payment from the customer. Any cameras located within the storeare typically present for security purposes.

SUMMARY

Shopping sessions in traditional stores may be associated with severalinefficiencies for both the customers and the store owners. For example,during busy periods within a store, a customer may spend a considerableamount of time waiting in line to pay the cashier for the items he/sheselected. The time spent waiting may even exceed the total amount oftime that the customer spent selecting the items. This may lead tocustomer frustration and potentially to a loss of repeat customerbusiness. As another example, traditional stores typically rely on thepresence of one or more employees to act as cashiers within the stores.Even when the store is otherwise empty, such employees are neverthelesspresent, in case a customer happens to enter the store to make apurchase. As a result, outside of peak business hours, much of acashier's time within a store may be spent idle.

This disclosure contemplates a virtual store tool that addresses one ormore of the above technical problems. The tool generates a virtual storeconfigured to emulate a physical store. The tool also generates a set ofvideos from camera feeds received from cameras located in the physicalstore, to track a customer during a shopping session in the physicalstore. In certain embodiments, the tool then uses the virtual store andthe videos of the shopping session in the physical store to generate avirtual shopping cart, storing a set of items configured to emulate theitems selected by the customer in the physical store. Accordingly, thetool may use the virtual shopping cart to charge the customer forhis/her purchases. In some embodiments, the tool may also be used inconjunction with an algorithm trained to determine the items selected bya customer during a shopping session in a physical store, based oninputs received from sensors located in the physical store. In suchembodiments, the tool uses the virtual store and the videos of theshopping session in the physical store to verify the determination madeby the algorithm. Certain embodiments of the tool are described below.

According to one embodiment, an apparatus includes an interface, adisplay, a memory, and a hardware processor communicatively coupled tothe memory and the display. The interface receives a first video feed.The first video feed includes a first camera feed corresponding to afirst camera located in a physical store and a second camera feedcorresponding to a second camera located in the physical store. Thefirst camera is directed at a first location in the physical store. Thesecond camera is directed at a second location in the physical store.The hardware processor stores a first video segment in the memory. Thefirst video segment is assigned to a first person and captures a portionof a shopping session of the first person in the physical storeoccurring during a time interval between a starting timestamp and anending timestamp. The first video segment includes a first camera feedsegment corresponding to a recording of the first camera feed from thestarting timestamp to the ending timestamp, and a second camera feedsegment corresponding to a recording of the second camera feed from thestarting timestamp to the ending timestamp. The processor also assigns afirst slider bar to the first video segment.

Playback of the first camera feed segment and the second camera feedsegment is synchronized and the first slider bar controls a playbackprogress of the first camera feed segment and the second camera feedsegment. The processor additionally displays the first camera feedsegment and a first copy of the first slider bar in a first region ofthe display. The processor further displays the second camera feedsegment and a second copy of the first slider bar in a second region ofthe display. The processor also receives an instruction from at leastone of the first copy of the first slider bar and the second copy of thefirst slider bar to adjust the playback progress of the first camerafeed segment and the second camera feed segment. In response toreceiving the instruction, the processor adjusts the playback progressof the first camera feed segment and the second camera feed segment.

According to another embodiment, an apparatus includes a display, aninterface, and a hardware processor communicatively coupled to thedisplay. The interface receives a rack camera feed from a rack cameralocated in a physical store. The rack camera is directed at a firstphysical rack of a set of physical racks located in the physical store.The hardware processor displays, in a first region of the display, avirtual layout of a virtual store. The virtual layout is configured toemulate a physical layout of the physical store. The virtual layoutincludes a first virtual rack assigned to a first physical rack and asecond virtual rack assigned to a second physical rack. Here, anarrangement of the first virtual rack and the second virtual rack in thevirtual layout is configured to emulate an arrangement of the firstphysical rack and the second physical rack in the physical layout.

The processor also receives an indication of an event associated withthe first physical rack. The event includes a person located in thephysical store interacting with the first physical rack. In response toreceiving the indication of the event associated with the first physicalrack, the processor displays, in a second region of the display, thefirst virtual rack. The first virtual rack includes a first virtualshelf and a second virtual shelf. The first virtual shelf includes afirst virtual item and the second virtual shelf includes a secondvirtual item. The first virtual item includes a graphical representationof a first physical item located on a first physical shelf of the firstphysical rack and the second virtual item includes a graphicalrepresentation of a second physical item located on a second physicalshelf of the first physical rack. The processor additionally displays,in a third region of the display, a rack video segment corresponding toa recording of the rack camera feed from a starting timestamp to anending timestamp. The rack video segment depicts the event associatedwith the first physical rack.

According to another embodiment, an apparatus includes a display, aninterface, and a hardware processor communicatively coupled to thedisplay. The interface receives a rack video from a rack camera locatedin a physical store. The rack camera is directed at a first physicalrack of a set of physical racks located in the physical store. The rackcamera captures video of the first physical rack during a shoppingsession of a person in the physical store. The processor displays, in afirst region of the display, a first virtual rack that emulates thefirst physical rack. The first virtual rack includes a first virtualshelf and a second virtual shelf. The first virtual shelf includes afirst virtual item and the second virtual shelf includes a secondvirtual item. The first virtual item includes a graphical representationof a first physical item located on a first physical shelf of the firstphysical rack and the second virtual item includes a graphicalrepresentation of a second physical item located on a second physicalshelf of the first physical rack.

The processor also displays, in a second region of the display, the rackvideo. The rack video depicts an event including the person interactingwith the first physical rack. The processor additionally displays, in athird region of the display, a virtual shopping cart. The processorfurther receives information associated with the event. The informationidentifies the first virtual item, and the rack video depicts that theperson selected the first physical item while interacting with the firstphysical rack. In response to receiving the information associated withthe event, the processor stores the first virtual item in the virtualshopping cart.

According to another embodiment, an apparatus configured to create avirtual layout of a virtual store to emulate a physical layout of aphysical store includes a memory and a hardware processorcommunicatively coupled to the memory. The hardware processor receives afirst physical position and a first physical orientation associated witha first physical rack located in the physical store. In response toreceiving the first physical position and the first physicalorientation, the processor places a first virtual rack at a firstvirtual position and with a first virtual orientation on the virtuallayout. The first virtual position of the first virtual rack on thevirtual layout represents the first physical position of the firstphysical rack on the physical layout and the first virtual orientationof the first virtual rack on the virtual layout represents the firstphysical orientation of the first physical rack on the physical layout.The processor also receives a first virtual item associated with a firstphysical item located on a first physical shelf of the first physicalrack. In response to receiving the first virtual item, the processorplaces the first virtual item on a first virtual shelf of the firstvirtual rack. The first virtual shelf of the first virtual rackrepresents the first physical shelf of the first physical rack.

The processor additionally receives a second virtual item associatedwith a second physical item located on a second physical shelf of thefirst physical rack. In response to receiving the second virtual item,the processor places the second virtual item on a second virtual shelfof the first virtual rack. The second virtual shelf of the first virtualrack represents the second physical shelf of the first physical rack.The processor further assigns a first rack camera located in thephysical store to the first virtual rack. The first rack camera capturesvideo that includes the first physical rack. The processor also storesthe virtual layout in the memory.

According to another embodiment, an apparatus includes a hardwareprocessor. The processor receives an algorithmic shopping cart thatincludes a first set of items. The first set of items is determined byan algorithm to have been selected by a first person during a shoppingsession in a physical store, based on a set of inputs received fromsensors located within the physical store. The processor also receives avirtual shopping cart that includes a second set of items associatedwith the shopping session. Video of the shopping session was captured bya set of cameras located in the physical store. The video depicts theperson selecting the second set of items. The processor additionallycompares the algorithmic shopping cart to the virtual shopping cart. Inresponse to comparing the algorithmic shopping cart to the virtualshopping cart, the processor determines that a discrepancy existsbetween the algorithmic shopping cart and the virtual shopping cart. Theprocessor further determines a subset of the set of inputs associatedwith the discrepancy. The processor also attaches metadata to thesubset. The metadata explains the discrepancy. The processoradditionally uses the subset to train the algorithm.

According to another embodiment, an apparatus includes a memory and ahardware processor communicatively coupled to the memory. The memorystores a first set of inputs, a first algorithmic shopping cart, andinstructions corresponding to a machine learning algorithm. The firstset of inputs includes information collected from sensors located in aphysical store during a shopping session of a first person in thephysical store. The first algorithmic shopping cart includes a first setof items. The first set of items was determined by an algorithm, basedon the first set of inputs, to have been selected by the first personduring the shopping session of the first person. The instructionscorresponding to the machine learning algorithm are configured, whenimplemented by the hardware processor, to use the first set of inputs toselect between using the first algorithmic shopping cart to process afirst transaction and using a first virtual shopping cart to process thefirst transaction. Here, the first transaction is associated with theshopping session of the first person and the first virtual shopping cartincludes items associated with the shopping session of the first person.

The processor uses the machine learning algorithm to determine, based onthe first set of inputs, to use the first algorithmic shopping cart toprocess the first transaction. Here, the first set of inputs areassociated with a first probability that the first algorithmic shoppingcart is accurate, and the first probability is greater than a threshold.In response to determining to use the first algorithmic shopping cart toprocess the first transaction, the processor also generates a firstreceipt based on the first algorithmic shopping cart. The first receiptincludes a first set of prices. Each price of the first set of pricescorresponds to an item of the first set of items. The processoradditionally sends the first receipt to the first person.

According to another embodiment, an apparatus includes a display and ahardware processor. The processor receives a refund request. The refundrequest includes a request for a refund of a price of an item charged toan account belonging to a person, and information identifying a shoppingsession of the person in a physical store. In response to receiving therefund request, the processor locates, using the information identifyingthe shopping session, a video segment of the physical store capturedduring the shopping session and stored in a database. The processoradditionally displays, in a first region of the display, the videosegment. The video segment depicts a scenario indicating that the persondid not select the item for purchase during the shopping session. Theprocessor further receives information indicating that the person didnot select the item for purchase during the shopping session. Theinformation is based at least in part on the video segment. In responseto receiving the information indicating that the person did not selectthe item for purchase during the shopping session, the processorprocesses the refund request, where processing the refund requestincludes crediting the account belonging to the person with the price ofthe item.

According to another embodiment, an apparatus includes a memory and asecond hardware processor communicatively coupled to the memory. Thememory stores instructions corresponding to a machine learningalgorithm. The machine learning algorithm is configured, whenimplemented by a first hardware processor, to use a set of inputs toselect between using an algorithmic shopping cart to process atransaction and using a virtual shopping cart to process thetransaction. The set of inputs includes information collected fromsensors located in a physical store during a shopping session of aperson in the physical store. The transaction is associated with theshopping session of the person. The algorithmic shopping cart includes afirst set of items determined by an algorithm, based on the set ofinputs, to have been selected by the person during the shopping session.The virtual shopping cart includes a second set of items associated withthe shopping session.

The second hardware processor receives feedback for a decision made bythe machine learning algorithm. The decision made by the machinelearning algorithm is at least one of a decision to use the algorithmicshopping cart to process the transaction, or a decision to use thevirtual shopping cart to process the transaction. The feedback indicateseither that the algorithmic shopping cart matches the virtual shoppingcart, or that the algorithmic shopping cart does not match the virtualshopping cart. The second processor also assigns a reward value to thefeedback. The reward value includes at least one of a first positivereward value, a second positive reward value, a first negative rewardvalue, or a second negative reward value. The reward value includes thefirst positive reward value, when the decision made by the machinelearning algorithm includes the decision to use the algorithmic shoppingcart to process the transaction and the feedback indicates that thealgorithmic shopping cart matches the virtual shopping cart. The rewardvalue includes the second positive reward value, when the decision madeby the machine learning algorithm includes the decision to use thevirtual shopping cart to process the transaction and the feedbackindicates that the algorithmic shopping cart does not match the virtualshopping cart. The reward value includes the first negative rewardvalue, when the decision made by the machine learning algorithm includesthe decision to use the algorithmic shopping cart to process thetransaction and the feedback indicates that the algorithmic shoppingcart does not match the virtual shopping cart. The reward value includesthe second negative reward value, when the decision made by the machinelearning algorithm comprises the decision to use the virtual shoppingcart to process the transaction and the feedback indicates that thealgorithmic shopping cart matches the virtual shopping cart. The secondprocessor additionally uses the reward value to update the machinelearning algorithm.

According to a further embodiment, an apparatus includes a display and ahardware processor communicatively coupled to the display. The processordisplays, in a first region of the display, a virtual shopping cart. Theprocessor also receives information indicating that an algorithmdetermined that a first physical item was selected by a person during ashopping session in a physical store. The algorithm determined that thefirst physical item was selected by the person based on a set of inputsreceived from sensors located within the physical store. In response toreceiving the information indicating that the algorithm determined thatthe first item was selected, the processor displays, in a second regionof the display, a first virtual item. The first virtual item includes agraphical representation of the first physical item. The processoradditionally displays, in a third region of the display, a first rackvideo captured during the shopping session of the person in the physicalstore. The first rack video was captured by a first rack camera of a setof rack cameras located in the physical store. The first rack camera isdirected at a first physical rack of a set of physical racks located inthe physical store. The first physical rack includes the first physicalitem. In response to displaying the first rack video, the processorreceives information identifying the first virtual item, where the firstrack video depicts that the person selected the first physical itemwhile interacting with the first physical rack. In response to receivingthe information identifying the first virtual item, the processor storesthe first virtual item in the virtual shopping cart.

Certain embodiments provide one or more technical advantages. Forexample, an embodiment reduces the processing resources spent whenreviewing surveillance video of a customer in a store, by presentingmultiple camera views of the store at once, synchronized with oneanother, and configured to capture the shopping session of the customer.As another example, an embodiment provides feedback for a machinelearning tracking algorithm, configured to track a customer in aphysical store, which may be used to improve the accuracy of the machinelearning algorithm. As another example, an embodiment provides animproved graphical user interface that enables a user to view multiplesurveillance videos on a single screen and to easily navigate amongstdifferent camera views. As a further example, an embodiment conservesprocessing resources by using a reinforcement learning algorithm todetermine that there is a high probability that certain determinations,made by a machine learning tracking algorithm that is configured toassign items to a customer by tracking the customer in a physical storeand identifying those items within the store with which the customerinteracts, are correct, such that no additional verification is needed.The system described in the present disclosure may particularly beintegrated into a practical application of a remote monitoring systemfor a physical location, such as a store, where inputs from sensorslocated in the store may be used to monitor and track events occurringwithin the store. In particular, the remote monitoring system is capableof automatically analyzing events occurring within the store andidentifying those events for which review by an external agent isdesirable (because, for example, the events are similar to previous onesin which review was deemed proper). In order to aid the external agentin reviewing an event, the system presents the agent with an improvedgraphical interface that is designed to display multiple video feedscaptured from inside the store during the event on a single screen, andwhich includes multiple controls to enable the agent to easily navigatethe different videos.

Certain embodiments may include none, some, or all of the abovetechnical advantages. One or more other technical advantages may bereadily apparent to one skilled in the art form the figures,descriptions, and claims included herein.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure, referenceis now made to the following description, taken in conjunction with theaccompanying drawings, in which:

FIGS. 1A and 1B present a comparison between a physical store and avirtual store;

FIGS. 2A and 2B present a comparison between a physical layout of aphysical store and a virtual layout of a virtual store;

FIGS. 3A and 3B present a comparison between a physical rack in aphysical store and a virtual rack in a virtual store;

FIG. 4 illustrates an example system according to the presentdisclosure;

FIG. 5A illustrates example locations in a physical store of camerasconfigured to capture regions of the store for use in the systemillustrated in FIG. 4;

FIG. 5B illustrates an example of the regions of a physical storecaptured by the layout cameras of the system illustrated in FIG. 4;

FIG. 6 illustrates the video processor component of the virtual storetool of the system illustrated in FIG. 4;

FIGS. 7A through 7C present an example illustrating the manner in whichthe virtual store tool of the system illustrated in FIG. 4 displayscamera feed segments associated with the layout cameras and the rackcameras of the system illustrated in FIG. 4;

FIG. 8 presents a flowchart illustrating the process by which thevirtual store tool of the system illustrated in FIG. 4 generates anddisplays camera feed segments associated with the layout cameras and therack cameras of the system illustrated in FIG. 4;

FIGS. 9A through 9D present examples illustrating the manner in whichthe virtual store tool of the system illustrated in FIG. 4 may virtuallyemulate a shopping session occurring in a physical store;

FIG. 10 presents a flowchart illustrating the manner in which thevirtual store tool of the system illustrated in FIG. 4 may virtuallyemulate a shopping session occurring in a physical store;

FIGS. 11A and 11B illustrate an example embodiment of a graphical userinterface generated by the virtual store tool of the system illustratedin FIG. 4, which may be used to generate a virtual layout configured toemulate a physical layout of a physical store;

FIG. 12 presents a flowchart illustrating the manner in which thevirtual store tool of the system illustrated in FIG. 4 may generate avirtual layout configured to emulate a physical layout of a physicalstore;

FIGS. 13A and 13B present examples of sensors that may be used toprovide input to an algorithm configured to determine items selected bya customer during a shopping session in a physical store;

FIGS. 13C and 13D illustrate an example of the use of sensors coupled toa physical shelf in a physical store to define zones of the physicalshelf and its corresponding virtual shelf;

FIG. 14 illustrates a resolution component of the virtual store tool ofthe system illustrated in FIG. 4;

FIG. 15 illustrates a machine learning component of the virtual storetool of the system illustrated in FIG. 4;

FIG. 16 presents a flowchart illustrating the manner by which thevirtual store tool of the system illustrated in FIG. 4 may providefeedback to an algorithm configured to determine the items selected by acustomer during a shopping session in a physical store;

FIGS. 17A and 17B present an example illustrating the use of the virtualstore tool of the system illustrated in FIG. 4 to virtually emulate ashopping session occurring in a physical store by using suggestions,provided by an algorithm, of products selected during the shoppingsession;

FIG. 18 presents a flowchart illustrating the use of the virtual storetool of the system illustrated in FIG. 4 to virtually emulate a shoppingsession occurring in a physical store by using suggestions, provided byan algorithm, of products selected during the shopping session;

FIGS. 19A through 19C present an example illustrating the use of thevirtual store tool of the system illustrated in FIG. 4 to process arefund request submitted in response to a prior virtual emulation by thevirtual store tool of a shopping session occurring in a physical store;

FIG. 20 presents a flowchart illustrating the use of the virtual storetool of the system illustrated in FIG. 4 to process a refund requestsubmitted in response to a prior virtual emulation by the virtual storetool of a shopping session occurring in a physical store;

FIG. 21 illustrates an example system according to the presentdisclosure, which includes a virtual store assistant which may be usedto identify shopping sessions as candidates for virtual emulation by thevirtual store tool;

FIG. 22 illustrates an example process by which a machine learningalgorithm, used by the virtual store assistant illustrated in FIG. 21,may decide whether or not a shopping session is a candidate for virtualemulation by the virtual store tool and subsequently receive feedbackregarding such decision; and

FIG. 23 presents a flowchart illustrating the use of the virtual storeassistant illustrated in FIG. 21 to decide whether or not a shoppingsession is a candidate for virtual emulation by the virtual store tooland subsequently receive feedback regarding such decision.

DETAILED DESCRIPTION

Embodiments of the present disclosure and its advantages may beunderstood by referring to FIGS. 1 through 16 of the drawings, likenumerals being used for like and corresponding parts of the variousdrawings. Additional information is disclosed in U.S. patent applicationSer. No. 16/663,663 entitled, “Scalable Position Tracking System ForTracking Position In Large Spaces”; U.S. patent application Ser. No.16/663,710 entitled, “Topview Object Tracking Using a Sensor Array”;U.S. patent application Ser. No. 16/664,470 entitled, “Customer-BasedVideo Feed”; U.S. patent application Ser. No. 16/664,490 entitled“System and Method for Presenting a Virtual Store Shelf that Emulates aPhysical Store Shelf”; U.S. patent application Ser. No. 16/938,676entitled “Feedback and Training for a Machine Learning AlgorithmConfigured to Determine Customer Purchases During a Shopping Session ata Physical Store”, which is a continuation of U.S. patent applicationSer. No. 16/794,083 entitled, “Feedback and Training for a MachineLearning Algorithm Configured to Determine Customer Purchases During aShopping Session at a Physical Store”, now U.S. Pat. No. 10,810,428,which is a continuation of U.S. patent application Ser. No. 16/663,564entitled, “Feedback and Training for a Machine Learning AlgorithmConfigured to Determine Customer Purchases During a Shopping Session ata Physical Store”, now U.S. Pat. No. 10,607,080; U.S. patent applicationSer. No. 17/021,011 entitled, “System and Method for Populating aVirtual Shopping Cart Based on Video of a Customer's Shopping Session ata Physical Store”, which is a continuation of U.S. patent applicationSer. No. 16/663,589 entitled, “System and Method for Populating aVirtual Shopping Cart Based on Video of a Customer's Shopping Session ata Physical Store”; and U.S. patent application Ser. No. 16/664,529entitled, “Tool for Generating a Virtual Store that Emulates a PhysicalStore”, which are all hereby incorporated by reference herein as ifreproduced in their entirety.

I. Introduction to Virtual Emulation

This disclosure is generally directed to generating a virtual store thatis configured to emulate a physical store, and using the virtual store,along with videos of a shopping session occurring within the physicalstore, to virtually emulate the physical shopping session. Although thisdisclosure describes virtual emulation of a physical store, thisdisclosure contemplates that any type of physical space (e.g., awarehouse, a storage center, an amusement park, an airport, an officebuilding, etc.) may be virtually emulated using the tool described inthe present disclosure. For example, the physical store may be aconvenience store or a grocery store. This disclosure also contemplatesthat the physical store may not be a physical building, but a physicalspace or environment in which shoppers may shop. For example, thephysical store may be a grab and go pantry at an airport, a kiosk in anoffice building, or an outdoor market at a park, etc.

As illustrated in FIG. 1A, a physical store 100 is a brick and mortarstore—i.e., a store that is located in a physical building. Customers105 (who may carry mobile devices 125) enter physical store 100 topurchase items. On the other hand, a virtual store 110 is a computerizedrepresentation of a physical store, displayed on a computer or otherdevice 115 belonging to a user 120, as illustrated in FIG. 1B. Thisdisclosure contemplates that user 120 may use virtual store 110 toemulate a shopping session of customer 105 in physical store 100.Virtual store 110 may be generated locally on device 115 or generatedremotely and transmitted over a network to device 115.

Virtual store 110 may be configured to emulate physical store 100 inseveral different ways. For example, in certain embodiments, and asillustrated in FIGS. 2A and 2B, the virtual layout 205 of virtual store110 is configured to emulate the physical layout 200 of physical store100. In particular, the shape, location, and orientation of virtualdisplay racks 230 a, 230 b, 230 c, and 230 d are configured to emulatethe shape, location, and orientation of physical display racks 210 a,210 b, 210 c, and 210 d. For example, in the example illustrated in FIG.2A, physical display racks 210 a and 210 b are located along back wall235 a of physical layout 200 of physical store 100. Accordingly, virtualdisplay racks 230 a and 230 b are placed along back wall 240 a ofvirtual layout 205 of virtual store 110, to emulate the location andorientation of physical display racks 210 a and 210 b. Similarly,virtual display rack 230 d is placed along side wall 240 b of virtuallayout 205, to emulate the position and orientation of physical displayrack 210 d along side wall 235 b, and virtual display rack 230 c isplaced in the center of virtual layout 205, to emulate the position andorientation of physical display rack 210 c.

As another example, in some embodiments, the contents of virtual displayracks 230 a, 230 b, 230 c, and 230 d are configured to emulate thecontents of physical display racks 210 a, 210 b, 210 c, and 210 d. Forexample, in certain embodiments, virtual display racks 230 a, 230 b, 230c, and 230 d are each assigned a list of items, wherein the list ofitems includes those items stored on physical rack 210 a, 210 b, 210 c,and 210 d, respectively. In other embodiments, each virtual display rackis assigned a set of virtual shelves, where the number and placement ofthe virtual shelves on the virtual display rack are configured toemulate the number and placement of the physical shelves on thecorresponding physical display rack. Each virtual shelf of the set ofvirtual shelves then holds a set of virtual items that is configured toemulate the set of physical items stored on a corresponding physicalshelf. Here the virtual items may be configured to emulate the physicalitems in terms of appearance and/or positioning on the virtual shelf.

As a specific example, FIGS. 3A and 3B present a comparison betweenphysical display rack 210 a and virtual display rack 230 a in oneembodiment. As seen in FIG. 3A, physical display rack 210 a includes twophysical shelves—first physical shelf 305 a and second physical shelf305 b. Accordingly, to emulate physical display rack 210 a, virtualdisplay rack 230 a also includes two shelves—first virtual shelf 310 aand second virtual shelf 310 b. Additionally, each of virtual shelves310 a and 310 b includes a set of virtual items configured to emulatethe physical items stored on the corresponding physical shelf ofphysical shelves 305 a and 305 b. For example, virtual shelf 310 aincludes first virtual item 320 a, located in first virtual zone 330 aof virtual shelf 310 a, second virtual item 320 b, located in secondvirtual zone 330 b of virtual shelf 310 a, and third virtual item 320 c,located in third virtual zone 330 c of virtual shelf 310 a, positionedto emulate the positioning of first physical item 315 a in firstphysical zone 325 a of physical shelf 305 a, second physical item 315 bin second physical zone 325 b of physical shelf 305 a, and thirdphysical item 315 c in third physical zone 325 c of physical shelf 305a. Similarly, virtual shelf 310 b includes fourth virtual item 320 d,fifth virtual item 320 e, and sixth virtual item 320 f, positioned,respectively, in fourth virtual zone 330 d, fifth virtual zone 330 e,and sixth virtual zone 330 f of virtual shelf 310 b, to emulate thepositioning of fourth physical item 315 d, fifth physical item 315 e,and sixth physical item 315 f in fourth physical zone 325 d, fifthphysical zone 325 e, and sixth physical zone 325 f of physical shelf 305b. Additionally, each of virtual items 320 a through 320 f is configuredto emulate the appearance of the corresponding physical item 315 a, 315b, 315 c, 315 d, 315 e, or 315 f. For example, each virtual item maycorrespond to a two-dimensional, graphical representation of thecorresponding physical item. In this manner, a virtual item may easilybe identified based on the appearance of its real world, physicalcounterpart.

II. System Overview

FIG. 4 illustrates an example system 400 that includes virtual storetool 405, device 115, display 410, network 430 a, network 430 b, layoutcameras 490, and rack cameras 495. In certain embodiments, system 400additionally includes external system 485 and sensors 498. Generally,virtual store tool 405 is configured to generate a virtual store 110that emulates a physical store 100. In certain embodiments, virtualstore tool 405 uses virtual store 110 to generate a receipt for ashopping session conducted by a person 105 in physical store 100, basedin part on videos tracking the shopping session, received from layoutcameras 490 and/or rack cameras 495 located in the physical store 100.In some embodiments, virtual store tool 405 uses virtual store 110 andvideos received from layout cameras 490 and/or rack cameras 495 toprocess a refund request submitted by a person 105 in response toreceiving a receipt from virtual store tool 405 for a shopping sessionconducted by person 105 in physical store 100. In some embodiments,virtual store tool 405 uses virtual store 110 and videos received fromlayout cameras 490 and rack cameras 495 to validate a determination madeby an algorithm 488 of the items selected by person 105 during theshopping session in physical store 100.

Device 115 includes any appropriate device for communicating withcomponents of system 400 over network 430 a. For example, device 115 maybe a telephone, a mobile phone, a computer, a laptop, a wireless orcellular telephone, a tablet, a server, an IoT device, and/or anautomated assistant, among others. This disclosure contemplates device115 being any appropriate device for sending and receivingcommunications over network 430 a. Device 115 may also include a userinterface, such as a microphone, keypad, or other appropriate terminalequipment usable by user 120. In some embodiments, an applicationexecuted by a processor of device 115 may perform the functionsdescribed herein.

Device 115 may include or be coupled to display 410. Display 410 is ascreen used by device 115 to display information received from virtualstore tool 405. In certain embodiments, display 410 is a standarddisplay used in a laptop computer. In certain other embodiments, display410 is an external display device connected to a laptop or desktopcomputer. In further embodiments, display 410 is a standard touch-screenliquid crystal display found in a typical smartphone or tablet.

As illustrated in FIG. 4, in certain embodiments, display 410 maypresent camera feed segments 415 a through 415 f, virtual layout 205,virtual rack 230, virtual shopping cart 420, and/or rack camera feedsegment 425. Camera feed segments 415 a through 415 f are videorecordings of camera feeds received by virtual store tool 405 fromlayout cameras 490 located in physical store 100, and are assigned to aperson 105 conducting a shopping session in physical store 100. Themethod by which virtual store tool 405 generates camera feed segments415 a through 415 f and displays camera feed segments 415 a through 415f on display 410 is described in further detail below, in the discussionof FIGS. 5 through 8.

Virtual layout 205 is assigned to the particular physical store 100 fromwhich virtual store tool 405 received the camera feeds associated withcamera feed segments 415 a through 415 f, and is configured to emulatethe physical layout 200 of that physical store. The method by whichvirtual store tool 405 generates virtual layout 205 is described infurther detail below, in the discussion of FIGS. 11 and 12.

Virtual rack 230 corresponds to one of the virtual racks included invirtual layout 205 and is configured to emulate a physical rack 210 ofphysical store 100. Accordingly, virtual rack 230 displays a set ofvirtual items 320, with each virtual item 320 representing a physicalitem 315 stored on the corresponding physical rack 210. Virtual shoppingcart 420 is used to hold virtual items 320, each of which represents aphysical item 315 selected by person 105 during the shopping session inphysical store 100. Rack camera feed segment 425 is a recording of acamera feed received by virtual store tool 405 from a rack camera 495.Rack camera 495 is directed at the physical rack 210 of physical store100 to which virtual rack 230 is assigned. For example, rack camera 495may be directed at the center of physical rack 210. Virtual shoppingcart 420 may be populated by virtual items 320 stored on virtual rack230, based in part on rack camera feed segment 425. The method by whichvirtual store tool 405 determines a virtual rack 230 to display ondisplay 410 and then uses virtual rack 230 to populate virtual shoppingcart 420 is described in further detail below, in the discussion ofFIGS. 9 and 10.

In some embodiments, and as described in further detail below, withrespect to FIGS. 11A and 11B, display 410 displays a graphical userinterface through which a user 120 may generate a virtual layout 205configured to emulate a physical layout 200 of a physical store 100. Incertain embodiments, and as described in further detail below, withrespect to FIGS. 17 and 18, display 410 displays a product list thatincludes items determined by algorithm 488 to have been selected byperson 105 during the shopping session in physical store 100. In someembodiments, and as described in further detail below, with respect toFIGS. 19 and 20, display 410 may display information associated with arequest for a refund of the price of one or more products charged to anaccount of person 105 during a prior shopping session in physical store100. For example, display 410 may display one or more virtual items 320which are the subjects of the refund request and/or a textualdescription of the reason person 105 has submitted the refund request.

Network 430 a facilitates communication between and amongst the variouscomponents of system 400 located outside of network 430 b, connectinglayout cameras 490, rack cameras 495, and external system 485 to virtualstore tool 405. This disclosure contemplates network 430 a being anysuitable network that facilitates communication between such componentsof system 400. Network 430 a may include any interconnecting systemcapable of transmitting audio, video, signals, data, messages, or anycombination of the preceding. Network 430 a may include all or a portionof a public switched telephone network (PSTN), a public or private datanetwork, a local area network (LAN), a metropolitan area network (MAN),a wide area network (WAN), a local, regional, or global communication orcomputer network, such as the Internet, a wireline or wireless network,an enterprise intranet, or any other suitable communication link,including combinations thereof, operable to facilitate communicationbetween the components.

Network 430 b facilitates communication between and amongst the variouscomponents of virtual store tool 405 and layout cameras 490, rackcameras 495, and external system 485. This disclosure contemplatesnetwork 430 b being any suitable network that facilitates communicationbetween the components of virtual store tool 405 and layout cameras 490,rack cameras 495, and external system 485. Network 430 b may include anyinterconnecting system capable of transmitting audio, video, signals,data, messages, or any combination of the preceding. Network 430 b mayinclude all or a portion of a public switched telephone network (PSTN),a public or private data network, a local area network (LAN), ametropolitan area network (MAN), a wide area network (WAN), a local,regional, or global communication or computer network, such as theInternet, a wireline or wireless network, an enterprise intranet, or anyother suitable communication link, including combinations thereof,operable to facilitate communication between the components. Thisdisclosure contemplates that network 430 b may be the same network asnetwork 430 a or a separate network from network 430 a.

As seen in FIG. 4, virtual store tool 405 includes a processor 435, amemory 440, and an interface 445. This disclosure contemplates processor435, memory 440, and interface 445 being configured to perform any ofthe functions of virtual store tool 405 described herein. Generally,virtual store tool 405 implements layout creator 460, video processor465, display controller 470, resolution component 475, and machinelearning module 480. Virtual store tool 405 may use layout creator 460to generate a virtual layout 205 configured to emulate a physical layout200 of a physical store 100. This function of virtual store tool 405 isdescribed in further detail below, in the discussion of FIGS. 11 and 12.Virtual store tool 405 may use video processor 465 to generate camerafeed segments 415 and rack camera feed segments 425, assigned to aperson 105 conducting a shopping session in physical store 100, based oncamera feeds received from layout cameras 490 and rack cameras 495,respectively. This function of virtual store tool 405 is described infurther detail below, in the discussion of FIGS. 5 through 8. Virtualstore tool 405 may use display controller 470 to adjust the informationdisplayed on display 410, based on input received from device 115. Thisfunction of virtual store tool 405 is described in further detail below,in the discussion of FIGS. 7 through 12, and 17 through 21. Virtualstore tool 405 may use resolution component 475 to compare the contentsof virtual cart 420 to an algorithmic shopping cart, determined by analgorithm 488 to contain items selected by customer 105 during ashopping session in physical store 100. Resolution component 475 mayidentify any discrepancies between virtual cart 420 and the algorithmiccart, resolve such discrepancies, and generate a receipt to send tocustomer 105. Resolution component 475 will be described in furtherdetail below, in the discussion of FIG. 14. Finally, virtual store tool405 may use machine learning module 480 to identify discrepanciesbetween virtual shopping cart 420 and the algorithmic cart and assignmetadata to the algorithmic inputs associated with the discrepancies.This metadata may then be used to retrain the algorithm. Machinelearning module 480 will be described in further detail below, in thediscussion of FIGS. 15 and 16.

Processor 435 is any electronic circuitry, including, but not limited tocentral processing units (CPUs), graphics processing units (GPUs),microprocessors, application specific integrated circuits (ASIC),application specific instruction set processor (ASIP), and/or statemachines, that communicatively couples to memory 440 and controls theoperation of virtual store tool 405. Processor 435 may be 8-bit, 16-bit,32-bit, 64-bit or of any other suitable architecture. Processor 435 mayinclude an arithmetic logic unit (ALU) for performing arithmetic andlogic operations, processor registers that supply operands to the ALUand store the results of ALU operations, and a control unit that fetchesinstructions from memory and executes them by directing the coordinatedoperations of the ALU, registers and other components. Processor 435 mayinclude other hardware and software that operates to control and processinformation. Processor 435 executes software stored on memory to performany of the functions described herein. Processor 435 controls theoperation and administration of virtual store tool 405 by processinginformation received from network 430 a, network 430 b, memory 440,device(s) 115, layout cameras 490, rack cameras 495, and external system485. Processor 435 may be a programmable logic device, amicrocontroller, a microprocessor, any suitable processing device, orany suitable combination of the preceding. Processor 435 is not limitedto a single processing device and may encompass multiple processingdevices.

Memory 440 may store, either permanently or temporarily, data,operational software, or other information for processor 435. Memory 440may include any one or a combination of volatile or non-volatile localor remote devices suitable for storing information. For example, memory440 may include random access memory (RAM), read only memory (ROM),magnetic storage devices, optical storage devices, or any other suitableinformation storage device or a combination of these devices. Thesoftware represents any suitable set of instructions, logic, or codeembodied in a computer-readable storage medium. For example, thesoftware may be embodied in memory 440, a disk, a CD, or a flash drive.In particular embodiments, the software may include an applicationexecutable by processor 435 to perform one or more of the functionsdescribed herein.

Additionally, in certain embodiments, memory 440 may store virtuallayouts 205 and sets of videos 450. Each of virtual layouts 205 athrough 205 n corresponds to a different physical store 100 and isconfigured to emulate the physical layout 200 of physical store 100.Virtual layouts 205 may be stored in memory 440 according to a storeidentification number. In this manner, a given virtual layout 205 a maybe retrieved from memory 440 using the store identification number. Thisdisclosure contemplates that set of videos 450 includes the camera feedsegments 415 and rack camera feed segments 425 assigned to a givenperson 105, for example, through identification number 455. Suchsegments are video recordings of camera feeds received by virtual storetool 405 from layout cameras 490 and rack cameras 495, respectively. Forexample, set of videos 450 may include camera feed segments 415 athrough 415 f and rack camera feed segments 425, assigned to a person105. The manner in which virtual store tool 405 generates sets of videos450 is described in further detail below, in the discussion of FIG. 6.

Interface 445 represents any suitable device operable to receiveinformation from networks 430 a and 430 b, transmit information throughnetworks 430 a and 430 b, perform suitable processing of theinformation, communicate to other devices, or any combination of thepreceding. For example, interface 445 receives camera feeds from layoutcameras 490 and rack cameras 495. As another example, interface 445receives input from device 115. Interface 445 represents any port orconnection, real or virtual, including any suitable hardware and/orsoftware, including protocol conversion and data processingcapabilities, to communicate through a LAN, WAN, or other communicationsystems that allows virtual store tool 405 to exchange information withdevice 115, layout cameras 490, rack cameras 495, and/or othercomponents of system 400 via networks 430 a and 430 b.

Database 482 stores videos 484 and shopping session identificationinformation 486. Each video of videos 484 a through 484 h is associatedwith a layout camera 490 or a rack camera 495 and corresponds to videocaptured by the associated camera 490/495 extending into the past over aperiod of time. This period of time may be the entire time cameras490/495 have been in operation, the past year, the past month, the pastweek, or any other suitable time period. For example, each of videos 484a through 484 f may correspond to all of the video captured over thepast year by layout cameras 490 a through 490 f, respectively, and eachof videos 484 g and 484 h may correspond to all of the video capturedover the past year by rack cameras 495 a and 495 b, respectively.Shopping session identification information 486 includes any informationthat may be used to identify those portions of videos 484 thatcorrespond to a specific shopping session of a specific customer 105and/or portions of a specific shopping session of a specific customer105. As an example, for each shopping session conducted in physicalstore 100, shopping session identification information 486 may includean identification number assigned to the shopping session and a startingand/or ending timestamp, where the starting and/or ending timestampidentifies the location of the shopping session within videos 484. Asanother example, for each shopping session conducted in physical store100, shopping session identification information 486 may include anidentification number assigned to the shopping session and a set oftimestamps and/or time intervals, each of which is associated with anitem. Each timestamp/time interval may indicate the approximate timewithin videos 484 at which it was determined (either by virtual storetool 405 or by algorithm 488) that the item associated with thetimestamp/time interval was selected for purchase during the shoppingsession. In certain embodiments, virtual store tool may use videos 484and/or shopping session identification information 486 to processrequests for refunds associated with previous shopping sessions inphysical store 100, as described in further detail below, in thediscussion of FIGS. 19 and 20.

External system 485 represents any system operable to receive input fromsensors 498 located in physical store 100 and to apply an algorithm 488to this input to track customers 105 in physical store 100 and/or todetermine physical items 315 selected by such customers during shoppingsessions in physical store 100. Embodiments of external system 485 aredescribed in U.S. patent application Ser. No. 16/663,710 entitled,“Topview Object Tracking Using a Sensor Array”, U.S. patent applicationSer. No. 17/104,889 entitled, “Food Detection Using a Sensor Array”, andU.S. patent application Ser. No. 17/104,925 entitled, “Self-ServeBeverage Detection and Assignment”, the contents of all of which areincorporated by reference herein. This disclosure contemplates thatsensors 498 may include any type of suitable sensors, located inphysical store 100, and operable to detect customers 105 in physicalstore 100. For example, physical store 100 may include cameras, lightdetection and range sensors, millimeter wave sensors, weight sensors,and/or any other appropriate sensors, operable to track a customer 105in physical store 100 and detect information associated with customer105 selecting one or more items 315 from physical store 100. Thisdisclosure also contemplates that algorithm(s) 488 may be any suitablealgorithm(s) for tracking customers 105 in physical store 100 anddetermining items 315 selected by customers 105. For example, in certainembodiments, algorithm(s) 488 may be a machine learning algorithm(s).

Layout cameras 490 and rack cameras 495 are located in physical store100. Each of layout cameras 490 a through 490 f is directed at alocation in physical store 100 and captures video and/or images of aregion in space around the location. Each of rack cameras 495 isdirected at a physical display rack 210 located in physical store 100and captures video and/or images of the physical display rack 210 andthe region in space around the physical display rack 210. Thisdisclosure contemplates that any number of layout cameras 490 may beinstalled in physical store 100 and connected to virtual store tool 405through network 430 b. Similarly, any number of rack cameras 495 may beinstalled in physical store 100 and connected to virtual store tool 405through network 430 b. For example, in some embodiments, physical store100 contains the same number of rack cameras 495 as physical shelves210. In other embodiments, physical store 100 contains more rack cameras495 than physical shelves 210. In certain embodiments, rack cameras 495are the same as layout cameras 490. In other embodiments, rack cameras495 are distinct from layout cameras 490. The operation of layoutcameras 490 and rack cameras 495 is described in further detail below,in the discussion of FIGS. 5 and 6.

Modifications, additions, or omissions may be made to the systemsdescribed herein without departing from the scope of the invention. Forexample, system 400 may include any number of users 120, devices 115,displays 410, networks 430 a and 430 b, layout cameras 490, rack cameras495, and external systems 485. The components may be integrated orseparated. Moreover, the operations may be performed by more, fewer, orother components. Additionally, the operations may be performed usingany suitable logic comprising software, hardware, and/or other logic.

III. Customer-Based Video Tracking

As described above, virtual store tool 405 may use virtual layout 205 toemulate a shopping session of a customer 105 in a physical store 100captured by cameras feed segments 415 and/or 425. FIGS. 5 through 8 areused to describe the method by which virtual store tool 405 generatesand displays camera feed segments 415 and/or 425.

a. Cameras Used for Customer-Based Video Tracking

FIG. 5A illustrates example locations of layout cameras 490 and rackcameras 495 in a physical store 100. The numbers of layout cameras 490and rack cameras 495 chosen for a physical store 100 may depend on thesize and/or layout of physical store 100. As seen in the example of FIG.5A, physical store 100 may include five layout cameras 490 a through 490e. While illustrated as located on the ceiling of physical store 100,this disclosure contemplates that layout cameras 490 may be mountedanywhere in physical store 100. Additionally, in the example of FIG. 5A,physical store 100 may include four rack cameras 495 a through 495 d.While illustrated as located both on the ceiling and sidewalls ofphysical store 100, this disclosure contemplates that rack cameras 495may be mounted anywhere in physical store 100. Rack cameras 495 may beseparate from layout cameras 490 or the same as layout cameras 490.

Each of rack cameras 495 is directed at a rack 210 located in physicalstore 100. For example, as illustrated in FIG. 5A, rack camera 495 a isdirected at physical display rack 210 a, rack camera 495 b is directedat physical display rack 210 b, rack camera 495 c is directed atphysical display rack 210 c, and rack camera 495 d is directed atphysical display rack 210 d. While FIG. 5A illustrates a set of fivelayout cameras 490 and a set of four rack cameras 495 in physical store100, this disclosure contemplates that any suitable number of layoutcameras 490 and rack cameras 495 may be used in physical store 100,depending on the size and/or layout of physical store 100. FIG. 5Aadditionally illustrates a set of turnstiles 510 located in physicalstore 100. Turnstiles 510 may be used to control the entry and exit ofcustomers 105 into or out of physical store 100, as described in furtherdetail below, in the discussion of FIG. 6.

As illustrated in FIG. 5B, each of layout cameras 490 is directed at aparticular location in physical store 100 and captures a region 505 ofthe layout 200 of physical store 100, surrounding the location. Forexample, first layout camera 490 a is directed at a first location andcaptures video and/or images of a first region 505 a of physical store100; second layout camera 490 b is directed at a second location andcaptures video and/or images of a second region 505 b of physical store100; third layout camera 490 c is directed at a third location andcaptures video and/or images of a third region 505 c of physical store100; fourth layout camera 490 d is directed at a fourth location andcaptures video and/or images of a fourth region 505 d of physical store100; and fifth layout camera 490 e is directed at a fifth location andcaptures video and/or images of a fifth region 505 e of physical store100. In certain embodiments, layout cameras 490 may capture overlappingregions of physical store 100. For example, as illustrated in FIG. 5B,all of third region 505 c is overlapped by portions of first region 505a, second region 505 b, fourth region 505 d, and fifth region 505 e. Theoverlapping regions of physical store 100 may be a result of theproximity of layout cameras 490 to one another. Generally, by capturingoverlapping regions of physical store 100, certain portions of physicallayout 200 can be captured by multiple layout cameras 490. This may bedesirable, to provide sufficient camera coverage of physical layout 200in the event that certain of layout cameras 490 malfunction or gooffline.

While illustrated in FIG. 5B as rectangular in shape, this disclosurecontemplates that regions 505 may be of any shape or size. For example,in certain embodiments, regions 505 are elliptical in shape. In someembodiments, regions 505 are of uniform size and shape. For example, asillustrated in FIG. 5B, regions 505 a through 505 e are all the sameshape and size. In other embodiments, regions 505 may include regions505 of different sizes and shapes.

b. Camera Feed Processing

The videos and/or images of physical store 100 captured by layoutcameras 490 and/or rack cameras 495 are transmitted to virtual storetool 405 in the form of camera feeds. Virtual store tool 405 then usesvideo processor 465 to generate camera feed segments 415 and rack camerafeed segments 425, assigned to a person 105 conducting a shoppingsession in physical store 100, based on these camera feeds. FIG. 6illustrates the operation of video processor 465 of virtual store tool405.

FIG. 6 presents an example of the operation of video processor 465 ofvirtual store tool 405, in an embodiment that includes a first layoutcamera 490 a, a second layout camera 490 b, and a rack camera 495 a. Asillustrated in FIG. 6, video processor 465 receives first camera feed605 a from first layout camera 490 a, second camera feed 605 b fromsecond layout camera 490 b, and rack camera feed 620 a from rack camera495 a. In certain embodiments, video processor 465 receives first camerafeed 605 a, second camera feed 605 b, and rack camera feed 620 adirectly from layout cameras 490 a, 490 b, and rack camera 495 a. Insome embodiments, video processor 465 receives first camera feed 605 a,second camera feed 605 b, and rack camera feed 620 a from interface 445.

Prior to processing camera feeds 605 a, 605 b, and 620 a, videoprocessor 465 first determines that a person 105, associated with anidentification number 455, entered physical store 100. This disclosurecontemplates that video processor 465 may determine that person 105entered physical store 100 in any suitable manner. For example, incertain embodiments, physical store 100 includes turnstiles 510, whichcontrol the entry of persons 105 into the store. A turnstile 510 mayopen upon person 105 scanning a QR code, located on a physical card or amobile device 125 belonging to person 105, using a scanner 515 attachedto the turnstile 510. Accordingly, the scanning of the QR code maygenerate a notification, sent to virtual store tool 405, indicating thatperson 105 entered physical store 100. As another example, in someembodiments, an algorithm 488 may be used to determine that person 105entered physical store 100, based on information received from sensors498 located in physical store 100. An example of such an algorithm 488will be described in further detail below, in the discussion of FIGS. 13through 16.

This disclosure contemplates that camera feeds 605 and 620 aresynchronized in terms of timestamps, such that video associated with agiven timestamp from each of camera feeds 605 a, 605 b, and 620 acorresponds to the same real time within physical store 100. Suchsynchronization may be achieved in any suitable manner. For example, incertain embodiments, layout cameras 490 and rack cameras 495 are pluggedinto the same ethernet switch. Determining that person 105 enteredphysical store 100 may then include receiving a starting timestamp 610corresponding to the timestamp at which person 105 entered physicalstore 100.

Given that data packets associated with first camera feed 605 a, secondcamera feed 605 b, and rack camera feed 620 a may arrive at virtualstore tool 405 over network 430 b at different times, this disclosurecontemplates that rather than virtual store tool 405 streaming firstcamera feed 605 a, second camera feed 605 b, and rack camera feed 620 afrom starting timestamp 610 onwards, video processor 465 of virtuallayout tool 405 stores recordings of first camera feed 605 a, secondcamera feed 605 b, and rack camera feed 620 a, lasting a predefinedamount of time, in memory 440. Such recordings may then be replayed,each synchronized with the others according to timestamps. Accordingly,once video processor 465 determines starting timestamp 610,corresponding to the timestamp at which person 105 entered physicalstore 100, video processor 465 next prepares segments of each camerafeed, starting at starting timestamp 610 and ending at ending timestamp615. Video processor 465 then stores these segments in memory 440. Forexample, video processor 465 prepares first camera feed segment 415 a,corresponding to a recording of first camera feed 605 a from startingtimestamp 610 to ending timestamp 615, second camera feed segment 415 b,corresponding to a recording of second camera feed 605 b from startingtimestamp 610 to ending timestamp 615, and rack camera feed segment 425a, corresponding to a recording of rack camera feed 620 a from startingtimestamp 610 to ending timestamp 615. Video processor 465 then storeseach of segments 415 a, 415 b, and 425 a in memory 450.

This disclosure contemplates that the time interval between startingtimestamp 610 and ending timestamp 615 may be any predetermined amountof time. For example, in certain embodiments, the time interval is fiveminutes. In order to capture video of a shopping session lasting morethan this predetermined amount of time, once camera feeds 605 a, 605 b,and 620 a reach ending timestamp 615, video processor 465 may storeadditional recordings of camera feeds 605 a, 605 b, and 620 a, startingat ending timestamp 615 and ending at a new ending timestamp, the newending timestamp occurring at the predetermined amount of time afterending timestamp 615. Video processor 465 may store any number ofadditional camera feed segments in memory 440, each corresponding to anadditional predetermined interval of time. In certain embodiments, videoprocessor 465 continues to record such additional camera feed segmentsuntil it receives an indication that person 105 has left physical store100.

Video processor 465 may store camera feed segments 415 and 425 for anynumber of persons 105. Accordingly, video processor 465 may store acollection of camera feed segments 415 and 425 assigned to a person 105as set of videos 450, where set of videos 450 is assigned identificationnumber 455 associated with person 105. As an example, a first person 105a may enter physical store 100 at a first starting timestamp 610 a and asecond person 105 b may enter physical store 100 at a second startingtimestamp 610 b after the first starting timestamp 610 a, wherein thesecond starting timestamp 610 b is within the predefined time intervalafter first starting timestamp 610 a, such that the camera feed segmentsrecorded for first person 105 a will contain video that overlaps withthe camera feed segments recorded for second person 105 b. Accordingly,video processor 465 may store the camera feed segments recorded forfirst person 105 a, along with an identification number 455 a, assignedto first person 105 a, in memory 440, as set of videos 450 a. Similarly,video processor 465 may store the camera feed segments recorded forsecond person 105 b, along with an identification number 455 b, assignedto second person 105 b, in memory 440, as set of videos 450 b. Virtualstore tool 405 may then retrieve from memory 440 the camera feedsegments associated with a given person 105, using the identificationnumber 455 assigned to that person.

Video processor 465 may be a software module stored in memory 440 andexecuted by processor 435. An example of the operation of videoprocessor 465 is as follows: (1) receive camera feeds 605 and 620 fromcameras 490 and 495, respectively; (2) determine that a person 105entered physical store 100; (3) determine the timestamp 610corresponding to the time at which person 105 entered physical store100; (4) record camera feed segments 415 and 425 from camera feeds 605and 620, respectively, where the camera feed segments correspond torecordings of camera feeds 605 and 620 from timestamp 610, correspondingto the time at which person 105 entered physical store 100, and lastinga predetermined amount of time to ending timestamp 615; and (5) storecamera feed segments 415 and 425 in memory 440 according to anidentification number 455 of person 105, as set of videos 450.

c. Displaying Camera Feed Segments

Once video processor 465 has recorded set of videos 450 from camerafeeds 605 and 620, virtual store tool 405 may then use displaycontroller 470 to display set of videos 450 on display 410 of device115. In certain embodiments, virtual store tool 405 may display set ofvideos 450 on display 410 of device 115 in the form of a graphical userinterface 700. FIGS. 7A through 7C present an example illustrating themanner in which virtual store tool 405 displays set of videos 450 ondisplay 410.

FIG. 7A illustrates an embodiment in which virtual store tool 405instructs display 410 to display four camera feed segments 415 a through415 d. Virtual store tool 405 displays first camera feed segment 415 ain a first region 702 a of display 410, second camera feed segment 415 bin a second region 702 b of display 410, third camera feed segment 415 cin a third region 702 c of display 410, and fourth camera feed segment415 d in a fourth region 702 d of display 410. Virtual store tool 405may instruct display 410 to display any number of camera feed segments415. For example, in certain embodiments, virtual display tool 405 mayinstruct display 410 to display the same number of camera feed segments415 as stored in set of videos 450. In some embodiments, virtual displaytool 405 may instruct display 410 to display fewer camera feed segments415 than stored in set of videos 450. This may be desirable inembodiments in which physical store 100 is a large store that includes alarge number of layout cameras 490. In such embodiments, displaying allof camera feed segments 415 on display 410 may make it difficult for auser 120 to view specific features of physical store 100 in any one ofthe displayed camera feed segments 415. Accordingly, virtual store tool405 may display a subset of camera feed segments 415 on display 410.Virtual store tool 405 may select a subset of camera feed segments 415to display on display 410 in any suitable manner. As an example, incertain embodiments, virtual store tool 405 may display a subset ofcamera feed segments 415 that includes, at any given time, those camerafeed segments 415 capturing regions of physical store 100 closest to thelocation of person 105, to whom set of videos 450 is assigned. In suchembodiments, when set of videos 450 depicts person 105 moving to a newlocation in physical store 100, virtual store tool 405 may replace thesubset of camera feed segments 415 currently displayed on display 410with a new subset of camera feed segments 415, which includes thosecamera feed segments 415 that capture regions of physical store 100closest to the new location of person 105. Virtual store tool 405 maydetermine the subset of camera feed segments 415 that capture regions ofphysical store 100 closest to the location or person 105 in any suitablemanner. For example, in certain embodiments, virtual store tool 405 mayreceive an indication of the location of person 105 from amachine-learning algorithm 488 configured to track the locations of aperson 105 in physical store 100, based on inputs received from a set ofsensors 498 located in physical store 100.

As illustrated in FIG. 7A, in addition to displaying camera feedsegments 415, virtual store tool 405 also assigns a slider bar 705 toset of videos 450 and displays copies of slider bar 705 along with eachcamera feed segment 415. For example, virtual store tool 405 displays afirst copy 705 a of slider bar 705 along with first camera feed segment415 a, a second copy 705 b of slider bar 705 along with second camerafeed segment 415 b, a third copy 705 c of slider bar 705 along withthird camera feed segment 415 c, and a fourth copy 705 d of slider bar705 along with fourth camera feed segment 415 d. Each copy of slider bar705 may contain a slider 710 configured to control the playback progressof the associated camera feed segment 415. For example, the position ofslider 710 on slider bar 705 indicates the current playback progress ofthe associated camera feed segment 415. The position of slider 710 maybe manually adjusted (e.g., by a user 120) to a new positioncorresponding to a new playback time. Such adjustment may result in theplayback of the associated camera feed segment adjusting to the newplayback time.

In certain embodiments, the playback of each camera feed segment 415 issynchronized with that of the other camera feed segments 415, such thatan adjustment of the slider 710 on any of the copies of slider bar 705leads to a corresponding adjustment of the playback progress of all ofthe displayed camera feed segments 415. For example, if slider 710 isadjusted on first copy 705 a of slider bar 705 from a first playbacktime to a second playback time, slider 710 on second copy 705 b ofslider bar 705, slider 710 on third copy 705 c of slider bar 705, andslider 710 on fourth copy 705 d of slider bar 705 will all similarlyadjust from the first playback time to the second playback time. Thismay be desirable for a user 120 using camera feed segments 415 toobserve a shopping session of a customer 105 in physical store 100. User120 may adjust the playback progress of camera feed segments 415 untiluser 120 determines that camera feed segments 415 have reached a pointof interest to user 120, rather than viewing the entire, uninterruptedplayback of camera feed segments 415.

In certain embodiments, slider bar 705 may include one or more markers715. For example, as illustrated in FIG. 7A, slider bar 705 may includea first marker 715 a, located at a first marker position on slider bar705 and corresponding to a first marker playback time, as well as asecond marker 715 b, located at a second marker position on slider bar705 and corresponding to a second marker playback time. First marker 715a is associated with a first event occurring at the first markerplayback time and second marker 715 b is associated with a second eventoccurring at the second marker playback time. The first event and thesecond event may include any type of events occurring within physicalstore 100. For example, the first event may be associated with a person105 a selecting a physical item 315 a from a physical shelf 305 alocated in a physical rack 210 a in physical store 100. Similarly, thesecond event may be associated with person 105 a selecting a secondphysical item 315 b from a second physical shelf 305 b located in asecond physical rack 210 b in physical store 100.

The locations for first marker 715 a and second marker 715 b on sliderbar 705 may be determined in any suitable manner. As an example, incertain embodiments, the first event, associated with first marker 715a, and the second event, associated with second marker 715 b, may bedetermined by an algorithm 488, based on a set of inputs received fromsensors 498 located within physical store 100. For example, algorithm488 may determine that the first event takes place at a first time,corresponding to a first timestamp, and that the second event takesplace at a second time, corresponding to a second timestamp. Virtualstore tool 405 may then use the first and second timestamps to placefirst marker 715 a and second marker 715 b on slider bar 705, atpositions corresponding to the timestamps. An example algorithm 488,used to determine the timing of the first and second events, isdescribed in further detail below, in the discussion of FIGS. 13 through16. The use of markers 715 may be desirable for a user 120 using camerafeed segments 415 to observe a shopping session of customer 105 inphysical store 100. Rather than viewing the entire, uninterruptedplayback of camera feed segments 415, user 120 may adjust the playbackprogress of camera feed segments 415 until slider 710 reaches one of theevents associated with first marker 715 a or second marker 715 b, to,for example, observe customer 105 selecting a physical item 315 from aphysical rack 210 in physical store 100.

As described above, in the discussion of FIG. 6, each of camera feedsegments 415 is of a predetermined time interval, lasting from astarting timestamp 610 to an ending timestamp 615. Accordingly, incertain embodiments in which customer 105 remains within physical store100 for longer than the predetermined time interval, multiple camerafeed segments may exist, from each of layout cameras 490. For example,virtual store tool 405 may store in memory 440 camera feed segments 415for a first time interval, a second time interval, a third timeinterval, and a fourth time interval. Memory 440 stores any number ofcamera feed segments 415 for any number of time intervals. In suchembodiments, when slider 710 reaches the end of slider bar 705, virtualstore tool 405 may replace those camera feed segments 415 currentlydisplayed on display 410, with the next set of camera feed segments 415,corresponding to the time interval immediately following the timeinterval captured by the currently displayed set of camera feed segments415. This process of replacing the currently displayed camera feedsegments 415 with a new set of camera feed segments 415, correspondingto the time interval immediately following the time interval captured bythe currently displayed set of camera feed segments 415 may continueuntil virtual store tool 405 determines that customer 105 has leftphysical store 100.

Virtual store tool 405 may determine that customer 105 has left physicalstore 100 in any suitable manner. As an example, in certain embodiments,virtual store tool 405 may determine that customer 105 has left physicalstore 100 based on input received from user 120. For example, inembodiments in which set of videos 450 are displayed on display 410 inthe form of a graphical user interface 700, the graphical user interface700 may include an interactive button 730 (e.g., an exit customerbutton) through which user 120 may indicate that he/she observedcustomer 105 exiting physical store 100, on camera feed segments 415, asillustrated in FIG. 7B. As another example, virtual store tool 405 maydetermine that customer 105 has left physical store 100 based oninformation received from an algorithm 488 configured to track customers105 within physical store 100. Such as algorithm 488 is described infurther detail below, in the discussion of FIGS. 13 through 16. As afurther example, virtual store tool 405 may determine that customer 105has left physical store 100 based on information received from physicalstore 100. For example, physical store 100 may include a set ofturnstiles 510 near the exit of physical store 100. In order to open aturnstile 510 and leave physical store 100, a customer 105 may be askedto scan the same QR code that he/she used to enter physical store 100.Scanning the QR code may then send a signal to virtual store tool 405,indicating that customer 105 has exited physical store 100.

In certain embodiments, a user 120 may request that virtual store tool405 display video segments 415/425 that include video of physical store100 from (1) before customer 105 entered physical store 100 and/or (2)after virtual store tool 405 determined that customer 105 exitedphysical store 100. As a specific example of a situation in which thismay arise, user 120 may view a second customer 105 b handing an item toa first customer 105 a, while viewing camera segments 415 a through 415f associated with the first customer 105 a. User 120 may not, however,have a clear view of which physical item 315 second customer 105 ahanded to first customer 105 b. Accordingly, user 120 may request thatvirtual store tool 405 display video of physical store 100 leading up tothe time interval captured by camera segments 415 a through 415 f, inorder to determine which physical item 315 that second customer 105 bmay have selected from physical store 100 to hand to first customer 105a. In certain embodiments, in response to receiving a request from user120 to display video segments 415/425 that include video of physicalstore 100 from before customer 105 entered physical store 100 and/orafter customer 105 exited physical store 100, virtual store tool 405 mayaccess videos 484 stored in database 482. As an example, in response toreceiving a request to generate video segments 415/425 that capturevideo of physical store 100 before customer 105 entered the store, videoprocessor 465 may identify the starting timestamp 610 associated withcustomer 105's shopping session, and locate this starting timestamp 610within videos 484 a through 484 h. Video processor 465 may then storesegments of each of videos 484 a through 484 h, beginning at a timestampcorresponding to a set time interval before starting timestamp 610 andlasting until starting timestamp 610, as video segments 415/425. Videoprocessor 465 may then display such segments 415/425 on graphical userinterface 700. As another example, in response to receiving a request togenerate video segments 415/425 that capture video of physical store 100after customer 105 exited the store, video processor 465 may identifythe ending timestamp 615 identified by virtual store tool 405 ascorresponding to when customer 105 exited physical store 100, and locatethis ending timestamp 615 within videos 484 a through 484 h. Videoprocessor 465 may then store segments of each of videos 484 a through484 h, beginning at ending timestamp 615 and lasting until a timestampcorresponding to a set time interval after ending timestamp 615, asvideo segments 415/425. Video processor 465 may then display suchsegments 415/425 on graphical user interface 700.

In certain embodiments, in order to assist a user 120 in determiningwhich of camera feed segments 415 may include information of interest tothe user, virtual store tool 405 is configured to highlight certaincamera feed segments 415, at certain times, based on events depicted inthose camera feed segments 415, at those certain times. For example, asillustrated in FIG. 7B, virtual store tool 405 may be configured todetermine that a given camera feed segment 415 a depicts customer 105 ata first time. Accordingly, virtual store tool 405 may highlight camerafeed segment 415 a in response to determining that slider 710 on sliderbar 705 reached that first time. Here, highlighting camera feed segment415 a may include any manner by which virtual store tool 405 may drawattention toward camera feed segment 415 a. For example, as illustratedin FIG. 7B, highlighting camera feed segment 415 a may include placing aframe 720 around camera feed segment 415 a. As another example,highlighting camera feed segment 415 a may include increasing the sizeof camera feed segment 415 a, depicted on display 410, relative to theother camera feed segments 415.

In certain embodiments, the graphical user interface 700 displayed ondisplay 410 may be used by a user 120 to monitor a shopping session of acustomer 105 a in physical store 100. To aid such a user 120 inmonitoring a particular customer 105 a in a physical store that includesseveral other customers 105, virtual store tool 405 may additionallydisplay an image 725 of customer 105 a, captured when customer 105 aentered physical store 100. For example, in certain embodiments in whichphysical store 100 includes turnstiles 510 to control the entry ofpersons 105 into the store, physical store 100 may include a cameraconfigured to take an image 725 of customer 105 a as customer 105 apasses through a turnstile 510.

In certain embodiments in which slider bar 705 includes one or moremarkers 715, each marker 715 may include metadata 740 describing theevent associated with the marker 715. An example of one such embodimentis illustrated in FIG. 7C. As described above, in the discussion of FIG.7A, each marker 715 a and 715 b may be associated with an eventconsisting of customer 105 a selecting a physical item 315 from aphysical shelf 305 of a physical rack 210 located in physical store 100.Accordingly, each marker may include metadata 740 indicating anidentification number 745 assigned to the physical item 315 selected bycustomer 105 a, an identification number 750 assigned to the physicalshelf 305 from which customer 105 a selected the physical item 315,and/or an identification number 755 assigned to the physical rack 210that includes the physical shelf 305 from which customer 105 a selectedthe physical item 315. In certain embodiments, item identificationnumber 745 may correspond to a zone identification number 745,identifying a zone of physical shelf 305 from which customer 105 aselected the physical item 315. The use of shelf zones will be describedin further detail below, in the discussion of FIGS. 13C and 13D.

Virtual store tool 405 may use metadata 740 in any suitable manner. Forexample, in certain embodiments, when slider 710 on slider bar 705reaches first marker 715 a, virtual store tool 405 may use metadata 740to determine that customer 105 selected a physical item 315 fromphysical rack 210. Accordingly, virtual store tool 405 may display rackcamera segment 425 a on display 410, where rack camera segment 425 adepicts video of physical rack 210. Rack camera segment 425 a may besynchronized with camera feed segments 415 a through 415 d, such that anadjustment of the slider 710 on any of the copies of slider bar 705leads to a corresponding adjustment of the playback progress of rackcamera segment 425 a. Automatically displaying rack camera segment 425a, in response to slider 710 reaching marker 715 on slider bar 705 maybe desirable, to provide a user 120 with a view of physical rack 210through which user 120 is able to observe customer 105 selecting aphysical item 315 from physical rack 210. In certain embodiments, user120 may be able to use a second graphical user interface to choose arack camera 495 from among several potential rack cameras 495 to assignto physical rack 210, to provide user 120 with a rack camera segment 425a that displays the best view of physical rack 210, as determined byuser 120. This aspect of virtual store tool 405 will be described infurther detail below, in the discussion of FIGS. 11 and 12.

FIG. 8 presents a flowchart illustrating the process by which virtualstore tool 405 generates camera feed segments 415 and 425 and displayssuch segments on display 410. In step 805, virtual store tool 405receives a set of layout camera feeds 605 from a set of layout cameras490 and a set of and rack camera feeds 620 from a set of rack cameras495 located in physical store 100. In step 810, virtual store tool 405determines whether a person 105 entered physical store 100. Thisdisclosure contemplates that virtual store tool 405 may determine thatperson 105 entered physical store 100 in any suitable manner. Forexample, in certain embodiments, physical store 100 includes turnstiles510, which control the entry of persons 105 into the store. A turnstile510 may be opened upon person 105 scanning a QR code, located on aphysical card or a mobile device 125 belonging to person 105.Accordingly, the scanning of the QR code may generate a notification,sent to virtual store tool 405, to indicate that person 105 enteredphysical store 100. As another example, in some embodiments, analgorithm 488 may be used to determine that person 105 entered physicalstore 100, based on information received from sensors 498 located inphysical store 100.

If, in step 810, virtual store tool 405 determines that person 105entered physical store 100, in step 815, virtual store tool 405 stores aset of camera feed segments 415 and 425 in memory 440. Each camera feedsegment of camera feed segments 415 corresponds to a recording of one ofthe camera feeds 605 from a starting timestamp 610 to an endingtimestamp 615. Similarly, each rack camera feed segment of rack camerafeed segments 425 corresponds to a recording of one of the rack camerafeeds 620 from starting timestamp 610 to ending timestamp 615. Startingtimestamp 610 corresponds to the time at which person 105 enteredphysical store 100. Ending timestamp 615 corresponds to a predeterminedtime interval after starting timestamp 610.

In step 820, virtual store tool 405 assigns copies of a slider bar 705to each camera feed segment 415 and 425. Slider 710 on each copy ofslider bar 705 moves forward as the corresponding camera feed segment415 and/or 425 progresses. In certain embodiments, the copies of sliderbar 705 are synchronized with one another such that all of camera feedsegments 415 and 425 progress together, at the same pace. Additionally,in such embodiments, an adjustment of the slider 710 on any of thecopies of slider bar 705 leads to a corresponding adjustment of theplayback progress of all of camera feed segments 415 and 425. This maybe desirable for a user 120 using camera feed segments 415 to observe ashopping session of a customer 105 in physical store 100. User 120 mayadjust the playback progress of camera feed segments 415 until user 120determines that camera feed segments 415 have reached a point ofinterest to user 120, rather than viewing the entire, uninterruptedplayback of camera feed segments 415.

In step 825, virtual store tool 405 presents one or more camera feedsegments 415 and/or 425 on display 410, along with corresponding copiesof slider bar 705. For example, virtual store tool 405 may display firstcamera feed segment 415 a, along with first copy 705 a of slider bar 705in a first region of display 410, second camera feed segment 415 b,along with second copy 705 b of slider bar 705 in a second region ofdisplay 410, third camera feed segment 415 c, along with third copy 705c of slider bar 705 in a third region of display 410, and fourth camerafeed segment 415 d, along with fourth copy 705 d of slider bar 705 in afourth region of display 410. Virtual store tool 405 additionally playscamera feed segments 415 and/or 425, such that slider 710 on each copyof slider bar 705 progresses.

In step 830, virtual store tool 405 next determines whether anadjustment occurred for any slider 710 in a copy of slider bar 705, froma first position on slider bar 705 to a second position on slider bar705, where the first position corresponds to a first playback time andthe second position corresponds to a second playback time. If, in step830, virtual store tool 405 determines that an adjustment occurred,virtual store tool 405 next adjusts the playback progress of each ofcamera feed segments 415 and 425 from the first playback time to thesecond playback time.

In step 840, virtual store tool 405 determines whether person 105 hasleft physical store 100. Virtual store tool 405 may determine thatcustomer 105 has left physical store 100 in any suitable manner. As anexample, in certain embodiments, virtual store tool 405 may determinethat customer 105 has left physical store 100 based on input receivedfrom user 120. For example, in embodiments in which camera feed segments415 and/or 425 are displayed on display 410 in the form of a graphicaluser interface 700, the graphical user interface 700 may include aninteractive button 730 (e.g., an exit customer button) through whichuser 120 may indicate that he/she observed customer 105 exiting physicalstore 100 on one or more camera feed segments 415. As another example,virtual store tool 405 may determine that customer 105 has left physicalstore 100 based on information received from an algorithm 488 configuredto track customers 105 within physical store 100. Such as algorithm 488is described in further detail below, in the discussion of FIGS. 13through 16. As a further example, virtual store tool 405 may determinethat customer 105 has left physical store 100 based on informationreceived from physical store 100. For example, physical store 100 mayinclude a set of turnstiles 510 near the exit of physical store 100. Inorder to open a turnstile 510 and leave physical store 100, a customer105 may be asked to scan the same QR code that he/she used to enterphysical store 100. Scanning the QR code may then send a signal tovirtual store tool 405, indicating that customer 105 has exited physicalstore 100.

If, in step 840, virtual store tool 405 determines that person 105 hasnot left physical store 100, in step 845, virtual store tool 405determines whether camera feed segments 415 and 425 have reached endingtimestamp 615. If, in step 845, virtual store tool 405 determines thatcamera feed segments 415 and 425 have not reached ending timestamp 615,virtual store tool returns to step 830, to determine whether anadjustment occurred for any slider 710 in a copy of slider bar 705, froma first position on slider bar 705 to a second position on slider bar705. On the other hand, if, in step 845, virtual store tool 405determines that camera feed segments 415 and 425 have reached endingtimestamp 615, virtual store tool 405 returns to step 825 and displays anew set of camera feed segments 415 and/or 425 on display 410, where thenew set of camera feed segments corresponds to recordings of camerafeeds 605 and/or 620 over a time interval immediately following theprevious time interval associated with the previous set of camera feedsegments 415 and/or 425.

Modifications, additions, or omissions may be made to method 800depicted in FIG. 8. Method 800 may include more, fewer, or other steps.For example, steps may be performed in parallel or in any suitableorder. While discussed as virtual store tool 405 (or components thereof)performing the steps, any suitable component of system 400, such asdevice(s) 115 for example, may perform one or more steps of the method.

IV. Virtual Emulation of a Shopping Session

As described above, camera feed segments 415 and 425 may be used inconjunction with virtual layout 205 in order to virtually emulate ashopping session occurring in physical store 100 and captured by camerafeed segments 415 and/or 425. For example, in certain embodiments,camera feed segments 415 and 425, along with virtual layout 205, may bepresented to a user 120, in the form of a graphical user interface 700.Here, camera feed segments 415 and 425 may be assigned to a customer 105and capture a shopping session of customer 105 in physical store 100.User 120 may monitor camera feed segments 415 and 425 to view customer120 selecting physical items 315 from physical racks 210. Accordingly,user 120 may populate a virtual shopping cart 420 with virtual items 320that represent the physical items 315 selected by customer 105, suchthat at the end of customer 105's shopping session, virtual shoppingcart 420 may include a virtual item 320 for each physical item 315selected by customer 105.

FIGS. 9A through 9D present further examples of a graphical userinterface 700, displayed on display 410, that may be used to virtuallyemulate a shopping session occurring in physical store 100 and capturedby camera feed segments 415 and 425. As illustrated in FIG. 9A, virtualstore tool 405 may display camera feed segments 415 in a first region955 of display 410, as described above in the discussion of FIGS. 7Athrough 7C. Virtual store tool 405 may additionally display virtuallayout 205 in a second region 960 of display 410. Virtual layout 205 isconfigured to emulate the physical layout 200 of physical store 100. Asillustrated in FIG. 9A, virtual layout 205 includes a set of virtualracks 230. This disclosure contemplates that virtual layout 205 mayinclude any number of virtual racks 230, where the number of virtualracks 230 displayed on virtual layout 205 corresponds to the number ofphysical racks 210 in physical store 100. The layout of virtual racks230 in virtual layout 205 is configured to emulate the arrangement ofthe corresponding physical racks 210 in physical store 100.

a. Receiving an Indication of an Event

As illustrated in FIG. 9B, virtual store tool 405 may receive anindication of an event associated with a physical rack 210 a located inphysical store 100. In certain embodiments, the event associated withphysical rack 210 a may include customer 105 interacting with physicalrack 210 a. For example, the event associated with physical rack 210 amay include customer 105 a approaching physical rack 210 a, and/orselecting a physical item 315 f from physical rack 210 a. The indicationof the event may include any suitable indication received by virtualstore tool 405. For example, in certain embodiments, the indication ofthe event may include user 120 selecting virtual shelf 230 a in virtuallayout 205, in response to viewing customer 105 approaching and/orinteracting with physical rack 210 a. As another example, the indicationof the event may include slider 710 on slider bar 705 reaching a marker715, where the marker 715 indicates the physical rack 210 associatedwith the event, through metadata 740. As a further example, in certainembodiments, the indication of the event may include receivinginformation from an algorithm 488 configured to determine that customer105 approached and/or selected an item 315 from physical rack 210 a,based on inputs received from sensors 498 located in physical store 100.

In certain embodiments, in which the graphical user interface 700displayed on display 410 may be used by a user 120 to monitor a shoppingsession of a customer 105 in physical store 100, virtual store tool 405may display a predicted location 950 of customer 105 on virtual layout205, based on the current playback progress of camera feed segments 415and/or 425. Predicted location 950 may correspond to the probablelocation of customer 105 in physical layout 200, as determined by analgorithm 488 configured to track customers 105 in physical store 100,based on inputs received from sensors 498 located in physical store 100,at a physical time corresponding to the current playback progress ofcamera feed segments 415 and/or 425. This may aid a user 120 inmonitoring a particular customer 105 a in a physical store that includesseveral other customers 105. While illustrated in FIG. 9B as dot 950 onvirtual layout 205, the predicted location of customer 105 may bepresented on virtual layout 205 in any suitable manner. For example, thepredicted location may be a line, including the predicted path ofcustomer 105. In such embodiments, the indication of the event mayinclude user 120 selecting virtual shelf 230 a in virtual layout 205, inresponse to viewing customer 105 approaching and/or interacting withphysical rack 210 a and/or viewing predicted location 950 of customer105 on virtual layout 205 indicating customer 105's proximity tophysical rack 210 a.

In response to receiving the indication of the event, virtual store tool405 may display the virtual rack 230 a corresponding to the physicalrack 210 a associated with the event, in a third region 905 of display410, where virtual rack 230 a is configured to emulate physical rack 210a. In certain embodiments, third region 905 of display 410 may belocated to the right of virtual layout 205. In certain embodiments,virtual store tool 405 may additionally highlight virtual rack 230 a, invirtual layout 205, in response to receiving the indication of the eventassociated with physical rack 210 a. Highlighting virtual rack 230 a mayinclude any method of distinguishing virtual rack 230 a from the othervirtual racks 230 b through 230 k. For example, as illustrated in FIG.9B, highlighting virtual rack 230 a may include placing a frame aroundvirtual rack 230 a. Highlighting virtual rack 230 a may additionallyinclude applying a color to virtual rack 230 a, and/or any othersuitable method of distinguishing virtual rack 230 a from the remainingvirtual racks 230 b through 230 k.

As illustrated in FIG. 9B, virtual rack 230 a, displayed in third region905 of display 410 includes a set of virtual items 320 a through 320 h.Virtual items 320 a through 320 h are configured to emulate the physicalitems stored on physical rack 210 a. In certain embodiments, virtualitems 320 a through 320 h are displayed in third region 905 as a list ofitems, where the names of the items in the list correspond to the namesof the physical items 315 a through 315 h stored on physical rack 210 a.In other embodiments, the appearance of virtual rack 230 a, displayed inthird region 905, is configured to emulate the appearance of physicalrack 210 a. For example, first virtual shelf 310 a is configured toemulate first physical shelf 305 a, second virtual shelf 310 b isconfigured to emulate second physical shelf 305 b, and third virtualshelf 310 c is configured to emulate third physical shelf 305 c. Inparticular, first virtual item 320 a is located in a first zone 330 a offirst virtual shelf 310 a to emulate the location of first physical item315 a in a first zone 325 a of first physical shelf 305 a. Similarly,second virtual item 320 b is located in a second zone 330 b of firstvirtual shelf 310 a, to the right of first virtual item 320 a, toemulate the location of second physical item 315 b in a second zone 325b of first physical shelf 305 a, and third virtual item 320 c is locatedin a third zone 330 c of first virtual shelf 310 a, to the right ofsecond virtual item 320 b, to emulate the location of third physicalitem 315 c in a third zone 325 c of first physical shelf 305 a. Virtualitems 320 d through 320 f are similarly located on second virtual shelf310 b to emulate the locations of the physical items 315 d through 315f, located on second physical shelf 305 b, and virtual items 320 g and320 h are located on third virtual shelf 310 c to emulate the locationsof physical items 315 g and 315 h located on third physical shelf 305 c.To further emulate physical items 315, each of virtual items 320 mayinclude a graphical representation of the corresponding physical item315.

In addition to displaying virtual rack 230 a in region 905 of display410, in response to receiving the indication of the event associatedwith physical rack 210 a, virtual store tool 405 may also display rackcamera segment 425 a in a fourth region 970 of display 410, asillustrated in FIG. 9C. In certain embodiments, the fourth region 970 ofdisplay 410 is to the right of third region 905. Rack camera segment 425a depicts physical rack 210 a, during the time interval in which theevent occurs. For example, in embodiments in which the event includescustomer 105 approaching physical rack 210 a, rack camera segment 425 adepicts customer 105 approaching physical rack 210 a. As anotherexample, in embodiments in which the event includes customer 105selecting an item 315 f from physical rack 210 a, rack camera segment425 a depicts customer 105 selecting item 315 f from physical rack 210a.

Rack camera segment 425 a may be synchronized with camera feed segments415 a through 415 f, such that an adjustment of the slider 710 on any ofthe copies of slider bar 705 leads to a corresponding adjustment of theplayback progress of rack camera segment 425 a. Displaying rack camerasegment 425 a, in response to receiving the indication of the event maybe desirable, to provide a user 120 with a view of physical rack 210 athrough which user 120 is able to observer customer 105 approachingand/or interacting with physical rack 210 a. For example, rack camerasegment 425 a may help user 120 to see if customer 105 selected an item315 from physical rack 210 a. User 120 may then use this information topopulate virtual cart 420, as described in further detail below, in thediscussion of FIG. 9D. In certain embodiments, user 120 may be able toselect a rack camera 495 to assign to physical rack 210 to provide user120 with a rack camera segment 425 a that displays the best view ofphysical rack 210 a, as determined by user 120. This aspect of virtualstore tool 405 will be described in further detail below, in thediscussion of FIGS. 11 and 12.

b. Receiving Information Identifying a Selected Item

In certain embodiments in which the event includes person 105 selectingan item from physical shelf 210 a, the indication of the event mayinclude information identifying the item selected by person 105. Forexample, if the event includes person 105 selecting physical item 315 ffrom physical rack 210 a, the indication of the event received byvirtual store tool 405 may include information identifying physical item315 f and/or virtual item 320 f. As an example, in certain embodiments,each physical shelf 305 of physical rack 210 a includes a set of weightsensors 1300, coupled to zones 325 of the physical shelf 305, asdescribed below, in the discussion of FIGS. 13B through 13D. When person105 removes an item 315 from physical shelf 305, the weight sensor 1300coupled to the zone 325 of physical shelf 305 on which the item 315 islocated may send information to virtual store tool 405 (either directly,or through other components of system 400, such as external system 485),indicating that the item 315 has been selected from physical shelf 305of physical rack 210 a. Virtual store tool 405 may use this informationto highlight the corresponding virtual item 320 on virtual rack 230 a,displayed in third region 905 of display 410. For example, a weightsensor coupled to a third zone of second physical shelf 305 b ofphysical rack 210 a may send information to virtual store tool 405indicating that item 315 f has been removed from the third zone ofsecond physical shelf 305 b of physical rack 210 a.

As another example, in certain embodiments, the indication of the eventmay include slider 710 on slider bar 705 reaching a marker 715. Markers715 may include metadata 740, as described above, in the discussion ofFIG. 7C. Metadata 740 may include information indicating anidentification number 745 assigned to the physical item 315 selected bycustomer 105, an identification number 750 assigned to the physicalshelf 305 from which customer 105 selected the physical item 315, and/oran identification number 755 assigned to the physical rack 210 thatincludes the physical shelf 305 from which customer 105 selected thephysical item 315. When, for example, slider 710 on slider bar 705reaches marker 715 a, virtual store tool 405 may read metadata 740assigned to marker 715 a, to identify that person 105 selected physicalitem 315 f from second physical shelf 305 b of physical rack 210 a.Markers 715 may be added to slider bar 705 in any suitable manner. Forexample, in certain embodiments, virtual display tool 405 adds markers715 to slider bar 705 based on information received from an algorithm488 configured to track customers 105 in physical store 100 and todetermine the physical items 315 selected by each customer 105, based oninputs received from sensors 498 located in physical store 100.

In response to receiving information identifying physical item 315 f asbeing the physical item selected by person 105 from physical rack 210 a,virtual store tool 405 may highlight sixth virtual item 320 f, locatedon second virtual shelf 310 b of virtual rack 230 a. Highlighting sixthvirtual item 320 f may include any method of distinguishing sixthvirtual item 320 f from the remaining virtual items 320. For example,highlighting sixth virtual item 320 f may include placing a frame aroundsixth virtual item 320 f, as illustrated in FIG. 9C, enlarging sixthvirtual item 320 f compared to the other virtual items 320, and/or anyother suitable method of distinguishing sixth virtual item 320 f fromthe remaining virtual items 320.

c. Populating a Virtual Cart

In certain embodiments, the graphical user interface 700 displayed byvirtual store tool 405 on display 410 may additionally include a virtualshopping cart 420, as illustrated in FIG. 9D. Virtual shopping cart 420may be used to further emulate a shopping session of a customer 105 inphysical store 100, by storing virtual items 320 corresponding to thephysical items 315 selected by person 105 during his/her shoppingsession. Virtual store tool 405 may display virtual shopping cart 420 ina fifth region 965 of display 410. In certain embodiments, the fifthregion 965 of display 410 is located between virtual rack 230 b,displayed in third region 905 of display 410, and rack camera segment425 a.

In certain such embodiments, receiving information identifying physicalitem 315 f as being the physical item selected by person 105 fromphysical rack 210 a, may include receiving information associated withdragging and dropping virtual item 320 f, corresponding to physical item315 f, from virtual rack 230 a, displayed in region 905, to virtualshopping cart 420. For example, a user 120 may observe customer 105selecting physical item 315 f on camera feeds segments 415 a through 415f and/or rack camera feed segment 425 a. Accordingly, user 120 mayselect virtual item 320 f from virtual rack 230 a, where virtual item320 f corresponds to physical item 315 f and is configured to emulatephysical item 315 f. User 120 may then drag virtual item 320 f tovirtual shopping cart 420 and drop virtual item 320 f in virtualshopping cart 420. In order to help aid user 120 in observing customer105 selecting a physical item 315 on camera feed segments 415 a through415 f and/or rack camera feed segment 425 a, in certain embodiments,user 120 can make any of the displayed camera feed segments 415 athrough 415 f and/or rack camera feed segment 425 a larger than theothers, by selecting the camera feed segments 415 a through 415 f and/orrack camera feed segment 425 a. For example, user 120 can click on agiven camera feed segment 415 or 425, to instruct virtual store tool 405to increase the size of the segment presented on display 410.

In response to receiving information identifying physical item 315 f asthe physical item selected by person 105 from physical rack 210 a—eitherfrom metadata 740, weight sensors 1300 coupled to physical shelf 305 b,a dragging and dropping of virtual item 320 f into virtual shopping cart420, and/or any other suitable method of receiving informationidentifying physical item 315 f—virtual store tool 405 may store virtualitem 320 f, corresponding to physical item 315 f, in virtual shoppingcart 420. Virtual shopping cart 420 may store any number of virtualitems 320. For example, as the playback of camera feed segments 415 and425 progresses, virtual store tool 405 may receive further informationidentifying an additional, different physical item 315 as having beenselected by person 105 from a physical rack 210. Physical rack 210 maybe the same as physical rack 210 a or different from physical rack 210a. In response to receiving the information identifying the additionalphysical item 315, virtual store tool 405 may store an additionalvirtual item 320, corresponding to the additional physical item 315, invirtual shopping cart 420. This process may repeat any number of times,such as a number of times corresponding to the number of times thecamera feed segments 415 and 425 indicate that a person 105 selected aphysical item 315 from a physical rack 210.

As illustrated in FIG. 9D, in certain embodiments, virtual shopping cart420 may display each virtual item 320 as a graphical representation ofthe corresponding physical item 315 and/or a textual description 910 ofthe corresponding physical item 315. Virtual shopping cart 420 may alsoindicate a quantity 915 of each virtual item 320 f contained in thevirtual shopping cart 420. For example, virtual shopping cart 420 mayindicate a quantity 915 of two virtual items 320 f, to emulate the factthat customer 105 selected two physical items 315 f from physical rack210 a. Quantity 915 of each virtual item 320 may be increased in anysuitable manner. For example, in certain embodiments, quantity 915 ofvirtual item 320 f may be increased by dragging and dropping virtualitem 320 f, corresponding to physical item 315 f, from virtual rack 230a, displayed in region 905, to virtual shopping cart 420 multiple times.As another example, in some embodiments, quantity 915 of virtual item320 f may be increased by a user 120 interacting with graphical userinterface 700 through an addition button 925. Similarly, quantity 915 ofvirtual item 320 f may be decreased by user 120 interacting withgraphical user interface 700 through a subtraction button 925. User 120may also remove virtual item 320 f from virtual shopping cart 420 byinteracting with graphical user interface 700 through a trash button930.

At the end of the shopping session of customer 105 in physical store 100(i.e., when virtual store tool 405 determines that customer 105 hasexited physical store 100), virtual shopping cart 420 may be used tocharge customer 105 for physical items 315 selected by customer 105during his/her shopping session, and to send a receipt to customer 105.Additionally, virtual shopping cart 420 may be used to validate adetermination made by an algorithm 488, based on inputs received fromsensors 498 located in physical store 100, of the physical items 315selected by customer 105 during his/her shopping session. These aspectsof virtual store tool 405 will be described in further detail below, inthe discussion of FIGS. 13 through 16.

d. Method for Virtually Emulating a Physical Shopping Session

FIG. 10 presents a flowchart illustrating the manner in which virtualstore tool 405 emulates a shopping session of a customer 105 in aphysical store 100, using virtual layout 205 and camera feed segments415 and/or 425 received from physical store 100, and capturing theshopping session. In step 1005, virtual store tool 405 displays virtuallayout 205 of virtual store 110. Virtual layout 205 is configured toemulate a physical layout 200 of physical store 100. In particular, thearrangement of virtual racks 230 on virtual layout 205 is configured toemulate the physical layout 200 of physical racks 210 in physical store100.

In step 1010, virtual store tool 405 determines whether the tool hasreceived an indication of an event associated with a person 105interacting with a physical rack 210 of physical store 100, during ashopping session in physical store 100. This event may include customer105 approaching a physical rack 210 and/or selecting a physical item 315from physical rack 210. The indication of the event may include anysuitable information that indicates that customer 105 interacted withphysical rack 210. For example, in certain embodiments, the indicationof the event may include user 120 selecting virtual shelf 230 in virtuallayout 205, in response to viewing customer 105 approaching and/orselecting physical item 315 from physical rack 210 on a set of camerafeed segments 415, generated from camera feeds 605 received from layoutcameras 490, located in physical store 100 and capturing the shoppingsession of customer 105. As another example, in certain embodiments, theindication of the event may include slider 710 on slider bar 705,assigned to camera feed segments 415, reaching a marker 715. Marker 715may include metadata 740 indicating the physical rack 210 associatedwith the event. As a further example, the indication of the event mayinclude receiving information from an algorithm 488 configured todetermine that customer 105 approached and/or selected an item 315 fromphysical rack 210, based on inputs received from sensors 498 located inphysical store 100.

If, in step 1010, virtual store tool 405 receives an indication of anevent associated with person 105 interacting with physical rack 210, instep 1015, virtual store tool 405 displays the virtual rack 230corresponding to physical rack 210 (i.e., configured to emulate physicalrack 210), in region 905 of display 410. Additionally, in step 1020,virtual store tool 405 displays a rack camera segment 425 generated froma rack camera feed 620 received from a rack camera 495 assigned tophysical rack 210. Rack camera segment 425 depicts physical rack 210during the time interval in which the event occurs.

In step 1025, virtual store tool 405 determines whether the tool hasreceived information identifying a first virtual item 320. As anexample, in certain embodiments, each physical shelf 305 of physicalrack 210 includes a set of weight sensors 1300, coupled to zones of thephysical shelf 305, as described below, in the discussion of FIGS. 13Bthrough 13D. When person 105 removes an item 315 from physical shelf305, the weight sensor 1300 coupled to the zone of physical shelf 305 onwhich the item 315 is located may send information to virtual store tool405 (either directly, or through other components of system 400, such asexternal system 485), indicating that the item 315 has been selectedfrom physical shelf 305 of physical rack 210 a. As another example, incertain embodiments, the indication of the event may include slider 710on slider bar 705 reaching marker 715 a or 715 b. Markers 715 a and 715b may include metadata 740, as described above, in the discussion ofFIG. 7C. Metadata 740 may include information indicating anidentification number 745 assigned to the physical item 315 selected bycustomer 105, an identification number 750 assigned to the physicalshelf 305 from which customer 105 selected the physical item 315, and/oran identification number 755 assigned to the physical rack 210 thatincludes the physical shelf 305 from which customer 105 selected thephysical item 315. Accordingly, when slider 710 on slider bar 705reaches a marker 715, virtual store tool 405 may receive informationidentifying physical item 315, by reading metadata 740 assigned tomarker 715, to identify that person 105 selected physical item 315 fromphysical shelf 305 of physical rack 210. Markers 715 may be added toslider bar 705 in any suitable manner. For example, in certainembodiments, virtual display tool 405 adds markers 715 to slider bar 705based on information received from an algorithm 488 configured to trackcustomers 105 in physical store 100 and to determine the physical items315 selected by each customer 105, based on inputs received from sensors498 located in physical store 100. As a further example, receivinginformation identifying physical item 315/virtual item 320 may includereceiving information associated with dragging and dropping virtual item320, configured to emulate physical item 315, from virtual rack 230,displayed in region 905 of display 410, to virtual shopping cart 420.

If, in step 1025, virtual store tool 405 determines that the tool hasreceived information identifying first virtual item 320/physical item315, in step 1030, virtual store tool 405 stores first virtual item 320in virtual shopping cart 420. In step 1035, virtual store tool 405determines whether the shopping session of customer 105 has ended (i.e.,whether customer 105 has left physical store 100). Virtual store tool405 may determine that customer 105 has left physical store 100 in anysuitable manner. As an example, in certain embodiments, virtual storetool 405 may determine that customer 105 has left physical store 100based on input received from user 120. For example, in embodiments inwhich camera feed segments 415 and/or 425 are displayed on a graphicaluser interface 700 on display 410, graphical user interface 700 mayadditionally include an interactive button 730 (e.g., an exit customerbutton) through which user 120 may indicate that he/she observedcustomer 105 exiting physical store 100, on one or more of camera feedsegments 415 and/or 425. As another example, virtual store tool 405 maydetermine that customer 105 has left physical store 100 based oninformation received from an algorithm 488 configured to track customers105 within physical store 100. As a further example, virtual store tool405 may determine that customer 105 has left physical store 100 based oninformation received from physical store 100. For example, physicalstore 100 may include a set of turnstiles 510 located near the exit ofphysical store 100. In order to open a turnstile 510 and leave physicalstore 100, a customer 105 may be asked to scan the same QR code thathe/she used to enter physical store 100. Scanning the QR code may thensend a signal to virtual store tool 405, indicating that customer 105has exited physical store 100. In certain embodiments, in response todetermining that customer 105 has left physical store 100, virtual storetool 105 sends a notification to a device 125 of customer 105,indicating that customer 105 should expect to receive a receipt forhis/her shopping session in physical store 105 within a set time period.

If, in step 1035, virtual store tool 405 determines that the shoppingsession of customer 105 in physical store 100 has not ended, virtualstore tool 405 returns to step 1010, to determine whether customer 105has selected any additional items 315 from physical racks 210.Specifically, virtual store tool 405 determines whether the tool hasreceived an indication of an event associated with customer 105interacting with another physical rack 210. Physical rack 210 may be thesame or a different physical rack from the physical rack with whichvirtual store tool 405 previously determined that customer 105interacted. In this manner, virtual store tool 405 may populate virtualcart 420 with any number of virtual items 320.

On the other hand, if, in step 1035, virtual store tool 405 determinesthat the shopping session has ended, then, in step 1040, virtual storetool 405 charges customer 105 for the items 315 selected by customer 105during the shopping session, based on the virtual items 320 stored invirtual cart 420, and generates a receipt. The manner in which virtualstore tool 405 generates the receipt is described in further detailbelow, in the discussion of FIG. 14. In order to charge customer 105,this disclosure contemplates that virtual store tool 405 may storepayment information for customer 105, according to an identificationnumber 455 assigned to customer 105, in memory 440. Next, in step 1045,virtual store tool 405 sends the receipt to customer 105.

Modifications, additions, or omissions may be made to method 1000depicted in FIG. 10. Method 1000 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as virtual store tool 405 (or componentsthereof) performing the steps, any suitable component of system 400,such as device(s) 115 for example, may perform one or more steps of themethod.

V. Virtual Layout Creation

In certain embodiments, layout creator 460 of virtual store tool 405 isconfigured to display a second graphical user interface 1100 throughwhich a user 120 may generate a virtual layout 205 configured to emulatea physical layout 200 of a physical store 100. FIGS. 11A and 11Billustrate an example embodiment of such a graphical user interface1100.

a. Placing Virtual Racks on Virtual Layout to Emulate the PhysicalLayout of Physical Racks

Layout creator 460 of virtual store tool 405 may generate a virtuallayout 205 configured to emulate a physical layout 200 of a physicalstore, in response to receiving a set of positions and orientationsassociated with physical racks 210 located in physical store 100. Layoutcreator 460 may receive the set of positions and orientations in anysuitable manner. For example, virtual store tool 405 may receive thepositions and orientations from user 120, through graphical interface1100, by user 120 creating virtual racks 230 on graphical interface 1100and then dragging and dropping the virtual racks 230 to given positionson virtual layout 205 and/or rotating virtual racks 230 to givenorientations on virtual layout 205. As another example, layout creator460 may receive the positions and orientations from a file uploaded tovirtual store tool 405. For example, user 120 may upload a fileincluding the positions and orientations using the “drop your file here”button 1150 on graphical user interface 1100. The file may include alist including pairs of positions and angles. In certain embodiments,each position may specify the center of mass position of a physicalshelf 210 in physical store 100. In some embodiments, each position mayspecify the position of a given corner of a physical shelf 210 inphysical store 100. The positions may be specified in terms of anycoordinate system superimposed on physical layout 200. For example, eachposition may be specified as an (x,y) coordinate of a Cartesiancoordinate system with an origin located in the middle of physical store100. In certain embodiments, each orientation may specify the angle of aphysical shelf 210 relative to a given direction. For example, eachorientation may specify the angle of a physical shelf 210 relative tothe x-axis of the Cartesian coordinate system of the previous example.In certain embodiments, for each physical shelf 210, the file mayadditionally include a length and width of the physical shelf 210.

In response to receiving the positions and orientations, layout creator460 places each virtual rack 230 at a virtual position and with avirtual orientation on virtual layout 205. Here, the virtual positionand the virtual orientation for a given virtual rack 230 on virtuallayout 205 represents the physical location and the physical orientationof the corresponding physical rack 210 in physical layout 200. WhileFIG. 11A illustrates an example including eleven virtual racks 230, thisdisclosure contemplates that virtual layout 205 may include any numberof virtual racks 230. Additionally, while FIG. 11A illustrates racks 230a through 230 k separated from one another on virtual layout 205, incertain embodiments, two or more racks may be touching one another. Forexample, as illustrated in FIG. 17B, a pair of racks 230 m and 230 maybe positioned back-to-back on virtual layout 205. Similarly, asillustrated in FIG. 17B, a pair of racks 230 b and 230 c may bepositioned side-by-side on virtual layout 205. In certain embodiments,virtual store tool stores the resulting virtual layout 205 in memory440, according to a store identification number 1105. Additionally,layout creator 460 may store each virtual rack 230 of virtual layout 205in memory 440 according to a rack identification number 755.

Virtual layout tool 405 may also modify a given virtual layout 205, inresponse to receiving a new position and/or orientation for any ofvirtual shelves 230 on virtual layout 205. Modifying virtual layout 205may be desirable in situations in which the physical layout 200 emulatedby virtual layout 205 has changed. Layout creator 460 may receive newpositions and/or new orientations for virtual shelves 230 in anysuitable manner. For example, layout creator 460 may read the newpositions and/or orientations from a file. The file may specify a newposition and/or orientation for a virtual rack 230 a using theidentification number 755 a assigned to virtual rack 230 a. For example,for each virtual rack 230, the file may include the identificationnumber 755 assigned to the virtual rack 230, Cartesian coordinates (x,y)of the new position for the rack, and an angle measured relative to thex-axis, specifying the new orientation for the rack. As another example,layout creator 460 may receive a new positions and/or orientation for avirtual rack 230, based on input received from graphical user interface1100. For example, as illustrated in FIG. 11A, layout creator 460 mayreceive input representing a dragging of virtual rack 230 k from a firstposition on virtual layout 205 to a new position 1130 on virtual layout205. In response to receiving such input, layout creator 460 may placevirtual rack 230 k at the new virtual position 1130, as illustrated inFIG. 11B. As another example, virtual store tool 205 may receive inputrepresenting a rotation of virtual rack 230 from a first orientation toa new orientation. In response to receiving such input, layout creator460 may place virtual rack 230 on virtual layout 205 with this neworientation.

b. Placing Virtual Items on Virtual Racks to Emulate the Physical ItemsLocated on Physical Racks

In addition to placing virtual racks 230 on virtual layout 205, layoutcreator 460 is operable to populate virtual racks 230 with virtual items320. For example, layout creator 460 may receive a planogram specifyingthe physical items 315 to be placed on each physical rack 210 inphysical store 100. For example, for each physical rack 210, theplanogram may include a list of physical items 315 to be placed on thephysical rack 210. For each physical item 315, the list may specify theshelf 305 of physical rack 210 on which the physical item 315 is to beplaced, as well as the zone 325 of each shelf 305 on which the physicalitem 315 is to be placed. In response to receiving the planogram, layoutcreator 460 may place corresponding virtual items 320 on virtual racks230. As another example, layout creator 460 may receive a list ofvirtual items 320 for each virtual rack 230, with each virtual item 320in the list associated with a physical item 315. Such a list may specifya store identification number 1105, a rack identification number 755, ashelf identification number 750, and/or a zone identification number 745for each virtual item 320 emulating a physical item 315. Here, storeidentification number 1105 identifies a physical store 100 storingphysical item 315, rack identification number 755 identifies a physicalrack 210 in physical store 100 holding physical item 315, shelfidentification number 750 identifies a physical shelf 305 of physicalrack 210, on which physical item 315 is placed, and zone identificationnumber 745 identifies a zone of physical shelf 305 housing physical item315. In certain embodiments, zone identification number 745 maycorrespond to a sensor identification number of a sensor 498 coupled tothe zone of physical shelf 305 housing physical item 315. Layout creator460 may then store the virtual item 320 in memory 440 according to storeidentification number 1105, rack identification number 755, shelfidentification number 750, and zone identification number 745, wherelayout creator 460 has assigned store identification number 1105 tovirtual layout 205, rack identification number 755 to virtual rack 230,shelf identification number 750 to virtual shelf 310, and zoneidentification number 745 to a virtual zone of virtual 310 configured toemulate the physical zone of physical shelf 305 housing physical item315. The division of physical shelves 305 and virtual shelves 310 intozones is described in further detail below, in the discussion of FIGS.13C and 13D.

As another example, layout creator 460 may receive virtual items 320 tostore on a given virtual rack 230 from a drop-down-menu that includes ascrollable list of items. An example of such a drop-down-menu 1135 isillustrated in FIG. 11B. As illustrated in FIGS. 11A and 11B, user 120may select a physical item name 1130 from drop-down-menu 1135 for agiven virtual shelf 230. In response, layout creator 460 may store thevirtual item 320 associated with the physical item 315 having physicalitem name 1130 in virtual shelf 230.

Second graphical user interface 1100 may also be used to assign rackcameras 495 to each of virtual racks 230 in virtual layout 205. Asillustrated in FIGS. 11A and 11B, layout creator 460 may present a setof rack camera feed segments 425 a through 425 f to user 120, throughsecond graphical user interface 1100. Each rack camera feed segment 425is generated from a rack camera feed 620 received from a rack camera 495located in physical store 100. In certain embodiments, a user 120 mayselect a rack camera 495 to assign to a virtual rack 230. User 120 mayselect a given rack camera 495 based on which of rack camera feedsegments 425 a through 425 f provides user 120 with the best view ofphysical rack 210 (emulated by virtual rack 230), as determined by user120. User 120 may select rack camera 495 in any suitable manner. As anexample, in certain embodiments, user 120 may assign a given rack camera495 to virtual rack 230 by clicking on the rack camera segment 425generated by rack camera 495 and displayed on second graphical userinterface 1100. For example, user 120 may click on rack camera segment425 a to assign rack camera 495 a, which generated rack camera segment425 a, to virtual rack 230 a. In response to user 120 clicking on rackcamera segment 425 a, layout creator 460 may associate a rack cameraidentification number 1125 a, assigned to rack camera 495 a, withvirtual rack 230 a.

Second graphical user interface 1100 may be used to generate any numberof virtual layouts 205, which layout creator 460 may store in memory 440according to store identification numbers 1105. Virtual store tool 405may later retrieve a given virtual layout 205 using the associated storeidentification number 1105 and display the virtual layout 205 on display410.

Layout creator 460 may be a software module stored in memory 440 andexecuted by processor 435. An example of the operation of layout creator460 is as follows: (1) receive a set of positions and orientationsassociated with physical racks 210 located in physical store 100; (2)for each received position and orientation, place a virtual rack 230 onvirtual layout 205, at a virtual position and with a virtual orientationrepresenting the physical position and physical orientation of thecorresponding physical rack 210 in physical layout 200; (3) if input isreceived, associated with a new position for a virtual rack 230, placevirtual rack 230 at the new position on virtual layout 205; (4) if inputis received, associated with a new orientation for a virtual rack 230,place virtual rack 230 on virtual layout 205, with the new orientation;(5) for each virtual rack 230, receive a set of virtual items 320, andplace the set of virtual items 320 on virtual rack 230; (6) for eachvirtual rack 230, assign a rack camera 495 to the virtual rack.

c. Method for Generating a Virtual Layout

FIG. 12 presents a flowchart illustrating the manner in which virtualstore tool 405 may generate a virtual layout 205 configured to emulate aphysical layout 200 of a physical store 100. In step 1205, virtual storetool 405 places a set of virtual racks 230 at virtual positions and withvirtual orientations on virtual layout 205, where the virtual positionsand the virtual orientations of virtual racks 230 are chosen to emulatethe physical positions and physical orientations of physical racks 210in physical store 100. Virtual store tool 405 may receive the virtualpositions and virtual orientations in any suitable manner. For example,virtual store tool 405 may receive the positions and orientation fromuser 120 through graphical user interface 1100. As another example,virtual store tool 405 may receive the positions and orientations from afile uploaded to virtual store tool 405.

In step 1210, virtual store tool 405 determines whether inputrepresenting a dragging of a virtual rack 230 to a new virtual positionwas received. If, in step 1210, virtual store tool 405 determines thatinput representing a dragging of a virtual rack 230 to a new virtualposition was received, in step 1215, virtual store tool 405 placesvirtual rack 230 at the new virtual position and proceeds to step 1220.On the other hand, if, in step 1210, virtual store tool 405 does notdetermine that input representing a dragging of a virtual rack 230 to anew virtual position was received, virtual store tool 405 simplyproceeds to step 1220.

In step 1220, virtual store tool 405 determines whether inputrepresenting a rotation of a virtual rack 230 from an initialorientation to a new orientation was received. If, in step 1220, virtualstore tool 405 determines that input representing a rotation of avirtual rack 230 from an initial orientation to a new orientation wasreceived, virtual store tool 405 adjusts the orientation of the virtualrack 230 from the initial orientation to the new orientation, in step1225, and proceeds to step 1230. On the other hand, if, in step 1220,virtual store tool 405 determines that input representing a rotation ofa virtual rack 230 from an initial orientation to a new orientation wasnot received, virtual store tool 405 proceeds to step 1230.

In step 1230, virtual store tool 405 receives, for each virtual rack230, a set of virtual items 320 assigned to the virtual rack 230.Virtual store tool 405 may receive the sets of virtual items 320 in anysuitable manner. For example, virtual store tool 405 may receive aplanogram, specifying the physical items 315 to be placed on eachphysical rack 210 in physical store 100. For example, for each physicalrack 210, the planogram may include a list of physical items 315 to beplaced on the physical rack. For each physical item 315, the list mayspecify the shelf 305 of physical rack 210 on which the physical item315 is to be placed, as well as the zone 325 of each shelf 305 on whichthe physical item 315 is to be placed. As another example, virtual storetool 405 may receive virtual items 320 from a drop-down-menu 1135displayed on display 410. The drop-down-menu 1135 may include a list ofphysical items 315 from which a user 120 may select one or more items tobe placed on each virtual rack 230. Drop-down-menu 1135 may include ascrollable list of any number of physical items 315. In response toreceiving a selection of a physical item 315 from drop-down-menu 1135,virtual store tool 405 may identify the corresponding virtual item 320.After virtual store tool 405 has received the sets of virtual items 320,in step 1235, virtual store tool 405 places each set of virtual items320 on the corresponding virtual rack 230.

In step 1240, virtual store tool 405 assigns a rack camera 495 to eachvirtual rack 230. Virtual store tool 405 may assign a rack camera 495 toeach virtual rack 230 in any suitable manner. For example, in certainembodiments, a user 120 may select rack cameras 495 to assign to virtualracks 230. User 120 may select a given rack camera 495 for a virtualrack 230 based on which of rack camera feed segments 425 a through 425 fprovides user 120 with the best view of the corresponding physical rack210, as determined by user 120. In step 1245, virtual store tool 405stores virtual layout 205 in memory 440. In certain embodiments, virtualstore tool 405 may store virtual layout 205 in memory 440 according to astore identification number 1105.

Modifications, additions, or omissions may be made to method 1200depicted in FIG. 12. Method 1200 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as virtual store tool 405 (or componentsthereof) performing the steps, any suitable component of system 400,such as device(s) 115 for example, may perform one or more steps of themethod.

VI. Use in Conjunction with an External Algorithm Configured to TrackCustomers in the Physical Store

Virtual store tool 405 may be used in conjunction with an algorithm 488,generated by external system 485, and configured to track customers 105and to determine items 315 selected by a given customer 105 a during ashopping session of customer 105 a in physical store 100, based oninputs received from sensors 498 located in physical store 100. Forexample, virtual store tool 405 may be used to validate thedeterminations made by algorithm 488 and/or to help improve the accuracyof algorithm 488. FIGS. 13 through 16 are used to describe this aspectof virtual store tool 405.

a. Algorithm Input Sensors

As described above, external algorithm 488 is configured to trackcustomers 105 and to determine items selected by a customer 105 during ashopping session in physical store 100, based on inputs received fromsensors 498 located in physical store 100. This disclosure contemplatesthat physical store 100 may include any type of suitable sensors 498.For example, physical store 100 may include cameras, light detection andrange sensors, millimeter wave sensors, weight sensors, and/or any otherappropriate sensors, operable to track a customer 105 in physical store100 and detect information associated with customer 105 selecting one ormore items from physical store 100.

FIGS. 13A through 13D present examples of an embodiment in whichphysical store 100 includes both cameras 1305 and weight sensors 1300for sensors 498. This disclosure contemplates that external system 485may process position information received from the cameras 1305, andweight information received from the weight sensors 1300, using analgorithm 488, to determine which customers 105 removed which items fromphysical display racks 210 located in physical store 100. In thismanner, external system 485 may generate an algorithmic shopping cart ofitems determined by the algorithm 488 to have been selected by acustomer 105, during a shopping session in physical store 100.

As seen in FIG. 13A, the interior of physical store 100 may include anarray of cameras 1305 positioned on the ceiling of store 100. In certainembodiments, this array of cameras 1305 may include layout cameras 490and/or rack cameras 495. In other embodiments, the array of cameras 1305is separate from layout cameras 490 and rack cameras 495. Generally, thearray of cameras 1305 produces videos of portions of the interior ofphysical store 100. These videos may include frames or images ofcustomers 105 within the space. External system 485 processes theseframes from array of cameras 1305 to detect customers 105 within theframes.

As illustrated in FIG. 13A, the array of cameras 1305 may includecameras 1305 arranged in a grid pattern across the ceiling of physicalstore 100. Although this disclosure shows the array of cameras 1305including fifty cameras 1305, the array of cameras 1305 may include anysuitable number of cameras 1305. Generally, cameras 1305 in the array ofcameras 1305 are arranged to form a rectangular array. In the example ofFIG. 13A, the array of cameras 1305 is a 5×10 array of cameras 1305(e.g., five rows and ten columns of cameras 1305). The array of cameras1305 may be arranged in an array of any suitable dimensions.

Each camera 1305 is communicatively coupled to external system 485 andcommunicates captured video to external system 485. Cameras 1305 arecommunicatively coupled to external system 485 in any suitable manner.For example, cameras 1305 may be hardwired to components of externalsystem 485. As another example, cameras 1305 may wirelessly couple toexternal system 485 using any suitable wireless protocol (e.g., WiFi).

Cameras 1305 may be any suitable devices for capturing videos of theinterior space of physical store 100. For example, cameras 1305 may bethree-dimensional cameras that can capture two-dimensional video of thespace (e.g., x-y plane) and also detect the heights of people and/orobjects in the video. As another example, cameras 1305 may betwo-dimensional cameras that capture two-dimensional videos of thespace. The array of cameras 1305 may include a mixture of differenttypes of cameras 1305.

FIG. 13B presents an example weight sensor 1300 that may be coupled to ashelf 305 of a physical rack 210 of physical store 100 to detect theweight of items 315 positioned on the shelf 305. Weight sensor 1300 maythen communicate this information to external system 485. Externalsystem 485 tracks the weights detected by weight sensors 1300 todetermine if, and when, items 315 are removed from the physical rack210.

As seen in FIG. 13B, weight sensor 1300 includes plates 1315 a and 1315b, load cells 1310 a, 1310 b, 1310 c, and 1310 d, and wires 1320 a, 1320b, 1320 c, 1320 d, and 1325. Generally, the components of weight sensor1300 are assembled so that weight sensor 1300 can detect a weight ofitems 315 positioned above or near weight sensor 1300.

Plates 1315 form surfaces that distribute the weight of items 315 acrossthe surfaces. Plates 1315 may be made of any suitable material, such as,for example, metal and/or plastic. Items 315 may be positioned above ornear plates 1315 and the weight of these items 315 may be distributedacross plates 1315.

Load cells 1310 are positioned between plates 1315 a and 1315 b. Loadcells 1310 produce electrical signals based on the weight experienced bythe load cells 1310. For example, load cells 1310 may be transducersthat convert an input mechanical force (e.g., weight, tension,compression, pressure, or torque) into an output electrical signal(e.g., current or voltage). As the input force increase, the outputelectrical signal may increase proportionally. Load cells 1310 may beany suitable type of load cell (e.g., hydraulic, pneumatic, and straingauge). Although load cells 1310 are illustrated as being cylindrical inshape, they may be any suitable size and shape that is appropriate forthe particular implementation contemplated.

The signals from load cells 1310 may be analyzed to determine an overallweight of items 315 positioned above or near weight sensor 1300. Loadcells 1310 may be positioned such that the weight of items 315positioned above or near weight sensor 1300 is evenly distributed toeach load cell 1310. In the example of FIG. 13B, load cells 1310 arepositioned substantially equidistant from corners of plates 1315 a and1315 b. For example, load cell 1310 a is positioned a distance d1 from acorner of plates 1315 a and 1315 b. Load cell 1310 b is positioned adistance d2 from a corner of plates 1315 a and 1315 b. Load cell 1310 cis positioned a distance d3 from a corner of plates 1315 a and 1315 b.Load cell 1310 d is positioned a distance d4 from a corner of 1315 a and1315 b. Distances d1, d2, d3 and d4 may be substantially equal to eachother. This disclosure contemplates distances differing by 5 to 10millimeters and still being considered substantially equal to eachother. By positioning load cells 1310 substantially equal distances fromcorners of plates 1315 a and 1315 b, the weight of items positionedabove or near weight sensor 1300 is evenly distributed across the loadcells 1310. As a result, the total weight of items positioned above ornear weight sensor 1300 can be determined by summing the weightsexperienced by the individual load cells 1310.

Load cells 1310 communicate electric signals that indicate a weightexperienced by the load cells 1310. For example, the load cells 1310 mayproduce an electric current that varies depending on the weight or forceexperienced by the load cells 1310. Each load cell 1310 is coupled to awire 1320 that carries the electric signal. In the example of FIG. 13B,load cell 1310 a is coupled to wire 1320 a; load cell 1310 b is coupledto wire 1320 b; load cell 1310 c is coupled to wire 1320 c; and loadcell 1310 d is coupled to wire 1320 d. Wires 1320 are grouped togetherto form wire 1325 that extends away from weight sensor 1300. Wire 1325carries the electric signals produced by load cells 1310 to a circuitboard that communicates the signals to external system 485.

In certain embodiments, and as illustrated in FIG. 13C, multiple weightsensors 1300 may be coupled to a given physical shelf 305 of physicalrack 210. For example, in the example illustrated in FIG. 13C, physicalshelf 305 includes four weight sensors 1300. The locations of weightsensors 1300 in physical shelf 305 may define a set of zones of physicalshelf 305. For example, first weight sensor 1300 a may define a firstzone 325 a, second weight sensor 1300 b may define a second zone 325 b,third weight sensor 1300 c may define a third zone 325 c, and fourthweight sensor 1300 d may define a fourth zone 325 d. In certainembodiments, each zone 325 may be associated with a different physicalitem 315, such that each weight sensor 1300 is configured to detectweight changes associated with the removal of a specific item 315 fromphysical shelf 305. Virtual shelves 310 may similarly be divided in aset of zones 330 a through 330 d, with each virtual zone 330 associatedwith a given virtual item 320, to emulate zones 325 a through 325 d ofphysical shelves 305. In this manner, when a signal is received from aweight sensor 1300 a, indicating the removal of a physical item 315 astored in first physical zone 325 a of physical shelf 305, the signalmay be used to identify virtual item 320 a, stored in first virtual zone330 a, based on the correspondence between first physical zone 325 a andfirst virtual zone 330 a.

b. Comparison Between Virtual Shopping Cart and Algorithmic ShoppingCart

In certain embodiments, virtual store tool 405 may be used inconjunction with an algorithm 488 trained to track customers 105 withinphysical store 100 and to determine the physical items 315 selected by acustomer 105 during a shopping session in physical store 100, based oninputs 498 received from sensors 498 located in physical store 100. Asdescribed above, in the discussion of FIGS. 13A through 13D, sensors 498may include cameras 1305 and weight sensors 1300.

Algorithm 488 may be programmed to determine the items 315 selected bycustomers 105 in physical store 100 in any suitable manner. For example,algorithm 488 may process video frames, received by external system 485from the array of cameras 1305, to determine coordinates for customers105 detected in the frames. Algorithm 488 may then timestamp thesecoordinates based on when the frames were received. Based on thecoordinates and the timestamps, algorithm 488 may determine thepositions of customers 105 in physical store 100. Algorithm 488 may alsoprocess signals received by external system 485 from weight sensors1300, to determine when items 315 were removed from physical shelves305. Using the positions of customers 105 in physical store 100, and thedeterminations of when items 315 were removed from physical shelves 305,algorithm 488 may determine which customers 105 took which items 315.

As an example of the use of virtual store tool 105 in conjunction withalgorithm 488, virtual store tool 405 may be used to resolvediscrepancies between the physical items 315 determined by algorithm 488to have been selected by customer 105 during a shopping session inphysical store 100 and the virtual items 320 determined by virtual storetool 405 to have been selected by customer 105 during the shoppingsession. When discrepancies exist between the physical items 315determined by algorithm 488 to have been selected by customer 105 andthe virtual items 320 determined by virtual store tool 405 to have beenselected by customer 105, the determination made by virtual store tool405 may also be used to improve the future accuracy of algorithm 488.FIGS. 14 through 16 are used to describe these aspects of virtual storetool 405.

FIG. 14 illustrates resolution component 475 of virtual store tool 405.Resolution component 475 is configured to compare virtual shopping cart420 to algorithmic shopping cart 1420. This disclosure contemplates thatvirtual store tool 405 may receive algorithmic shopping cart 1420 fromexternal system 485. Algorithmic shopping cart 1420 may include physicalitems 315 determined by an algorithm 488 to have been selected bycustomer 105, based on inputs received from sensors 498 (includingcameras 1305 and weight sensors 1300) located in physical store 100. Forexample, algorithmic shopping cart may include first physical item 315l, second physical item 315 m, and third physical item 315 n. Each ofphysical items 315 l through 3135 n is associated with a given purchaseprice. For example, first physical item 315 l is associated with a firstphysical price 1440, second physical item 315 m is associated with asecond physical price 1450, and third physical item 315 n is associatedwith a third physical price 1460. While illustrated in FIG. 14 asincluding three physical items 315, this disclosure contemplates thatalgorithmic shopping cart 1420 may include any number of physical items315. Virtual shopping cart 420 includes first virtual item 320 i, secondvirtual item 320 j, and third virtual item 320 k, each determined byvirtual store tool 405 to have been selected by customer 105 during ashopping session in physical store 100. Each of virtual items 320 ithrough 320 k is associated with a given purchase price. For example,first virtual item 320 i is associated with a first virtual price 1410,second virtual item 320 j is associated with a second virtual price1420, and third virtual item 320 k is associated with a third virtualprice 1425. While illustrated in FIG. 14 as including three virtualitems 320, this disclosure contemplates that virtual shopping cart 420may include any number of virtual items 320. Furthermore, virtualshopping cart 420 need not include the same number of items asalgorithmic shopping cart 1420.

Resolution component 475 is configured to perform a comparison 1430between the contents of virtual shopping cart 420 and the contents ofalgorithmic shopping cart 1420 to determine if any discrepancies 1435exist between the two carts. Resolution component 475 may determineeither that: (1) the two carts are consistent with one another; or (2)the two carts are inconsistent with one another. In certain embodiments,in response to determining that the two carts are inconsistent with oneanother, resolution component 475 may determine that (1) the two cartsare inconsistent with one another because virtual cart 420 includes anerror; or (2) the two carts are inconsistent with one another, becausealgorithmic cart 1420 includes an error.

Determining that the two carts are consistent with one another mayinclude determining that first virtual item 320 i, present in virtualshopping cart 420, is configured to emulate first physical item 315 l,which is present in algorithmic shopping cart 1420, second virtual item320 j, present in virtual shopping cart 420, is configured to emulatesecond physical item 315 m, which is present in algorithmic shoppingcart 1420, and third virtual item 320 k, present in virtual shoppingcart 420, is configured to emulate third physical item 315 n, which ispresent in algorithmic shopping cart 1420. On the other hand,determining that the two carts are inconsistent with one another mayinclude: (1) determining that virtual shopping cart 420 includes morevirtual items 320 than algorithmic shopping cart 1420 includes physicalitems 315; (2) determining that virtual shopping cart 420 includes fewervirtual items 320 than algorithmic shopping cart 1420 includes physicalitems 315; (3) determining that a virtual item 320, present in virtualshopping cart 420, is configured to emulate a physical item 315, whichis not present in algorithmic shopping cart 1420; or (4) determiningthat no virtual item 320, present in virtual shopping cart 420, isconfigured to emulate a physical item 315, present in algorithmicshopping cart 1420.

This disclosure contemplates that in embodiments in which resolutioncomponent 475 may determine that the two carts are inconsistent with oneanother because one of the two carts includes an error, resolutioncomponent 475 may determine that one of the two carts includes an errorin any suitable manner. As an example, in certain embodiments,resolution component 475 may always determine that algorithmic shoppingcart 1420 includes an error any time a discrepancy exists betweenvirtual shopping cart 420 and algorithmic shopping cart 1420. As anotherexample, resolution component 475 may determine that one of the cartsincludes an error, based on the type of discrepancy 1435 that existsbetween virtual cart 420 and algorithmic cart 1420. For example,resolution component 475 may be configured to determine that virtualshopping cart 420 includes an error when the discrepancy 1435 betweenthe two carts is a result of differing quantities of a particular itembetween the two carts. For instance, virtual cart 420 may include oneinstance of first virtual item 320 i, configured to emulate firstphysical item 315 l, while algorithmic shopping cart 1420 may includetwo instances of first physical item 315 l. In such situations, it maybe more likely that virtual shopping cart 420 includes an incorrectquantity of first virtual item 320 i, than algorithmic shopping cart1420 includes an incorrect quantity of first physical item 315 l, as itmay be difficult to tell from camera feed segments 415 and/or 425 thatcustomer 105 selected more than one physical item 315 l from a givenphysical shelf 305. On the other hand, the information received fromweight sensors 1300 in physical store 100, may more accurately indicatethat more than one physical item 315 l was selected from physical shelf305. For discrepancies 1435 that do not include differing quantities ofa particular item between the two carts, resolution component 475 may beconfigured to determine that algorithmic shopping cart 1420 includes anerror, as a default.

As another example, resolution component 475 may be configured todetermine that either virtual shopping cart 420 or algorithmic shoppingcart 1420 includes an error based on input received from user 120. Forexample, in response to determining that a discrepancy 1435 existsbetween virtual shopping cart 420 and algorithmic shopping cart 1420,resolution component 475 may send a message to device 115, advising user120 of the discrepancy 1435. User 120 may then send a response tovirtual store tool 405 indicating either that virtual shopping cart 420includes an error, or that algorithmic shopping cart 1420 includes anerror. User 120 may determine that one of virtual shopping cart 420 andalgorithmic shopping cart 1420 include an error in any suitable manner.As an example, user 120 may review camera feed segments 415 and/or 425to either confirm the contents of virtual shopping cart 420 or determinethat virtual shopping cart 420 includes an error. For example, if thediscrepancy includes an absence of a physical item 315 from algorithmicshopping cart 1420, where the corresponding virtual item 320 is presentin virtual shopping cart 420, user 120 may review camera feed segments415 and/or 425 to confirm that the camera feed segments capture customer105 selecting the physical item 315 from a physical rack 210. As anotherexample, if the discrepancy includes the presence of a physical item 315in algorithmic shopping cart 1420, where the corresponding virtual item320 is absent from virtual shopping cart 420, user 120 may review camerafeed segments 415 and/or 425 to either (1) confirm that user 120 neverobserves customer 105 selecting the physical item 315 from a physicalrack 210; or (2) confirm that while user 120 views customer 105selecting the physical item 315 from a physical rack 210 in camera feedsegments 415 and/or 425, user 120 subsequently views the customer 105setting down the physical item 315 and leaving the physical store 100with the item 315.

Resolution component 475 is also configured to generate a receipt 1465for customer 105. In certain embodiments, resolution component 475generates receipt 1465 based on the contents of virtual shopping cart420. For example, resolution component 475 may generate receipt 1465based on the contents of virtual shopping cart 420 before performingcomparison 1430. In other embodiments, resolution component 475 maygenerate receipt 1465 based on comparison 1430. For example, ifresolution component 475 determines that virtual shopping cart 420 isconsistent with algorithmic shopping cart 1420, resolution component 475may generate receipt 1465 a for customer 105. Receipt 1465 a may includethe total cost 1475 of first virtual item 320 i, second virtual item 320j, and third virtual item 320 k, as determined from first virtual price1410, second virtual price 1420, and third virtual price 1425. Here,since virtual cart 420 is consistent with algorithmic shopping cart1420, determining the total cost 1475 of first virtual item 320 i,second virtual item 320 j, and third virtual item 320 k is equivalent todetermining the total cost of first physical item 315 l, second physicalitem 315 m, and third physical item 315 n. As another example, ifresolution component 475 determines that virtual shopping cart 420includes an error, resolution component 475 may generate receipt 1465 bfor customer 105. Receipt 1465 b may include the total cost 1480 offirst physical item 315 l, second physical item 315 m, and thirdphysical item 315 n, as determined from first physical price 1440,second physical price 1450, and third physical price 1460. As a furtherexample, if resolution component 475 determines that algorithmicshopping cart 1420 includes an error, resolution component 475 maygenerate receipt 1465 c for customer 105. Receipt 1465 c may include thetotal cost 1475 of first virtual item 320 i, second virtual item 320 j,and third virtual item 320 k, as determined from first virtual price1410, second virtual price 1420, and third virtual price 1425. Onceresolution component 475 has generated a receipt 1465 for customer 105,resolution component 475 may charge customer 105, based on receipt 1465,and/or send receipt 1465 to device 125, belonging to customer 105.

Resolution component 475 may be a software module stored in memory 440and executed by processor 435. An example of the operation of resolutioncomponent 475 is as follows: (1) receive virtual shopping cart 420 andalgorithmic shopping cart 1420; (2) determine if the number of virtualitems 320 in virtual shopping cart 420 is the same as the number ofphysical items 315 in algorithmic shopping cart 1420; (3) if the numberof virtual items 320 in virtual shopping cart 420 is different from thenumber of physical items 315 in algorithmic shopping cart 1420, labelthe two carts as inconsistent; (4) if the number of virtual items 320 invirtual shopping cart 420 is the same as the number of physical items315 in algorithmic shopping cart 1420, determine if virtual shoppingcart 420 includes any virtual items 320 for which algorithmic shoppingcart 1420 does not include a corresponding physical item 315; (5) ifvirtual shopping cart 420 includes any virtual items 320 for whichalgorithmic shopping cart 1420 does not include a corresponding physicalitem 315, label the two carts as inconsistent; (6) if virtual shoppingcart 420 does not include any virtual items 320 for which algorithmicshopping 1420 does not include a corresponding physical item 315, labelthe two carts as consistent; (7) if the two carts are labelled asconsistent generate receipt 1465 a; (8) if the two carts are labelled asinconsistent, determine whether virtual cart 420 includes an error; (9)if virtual cart 420 includes an error, generate receipt 1465 b; (10) ifvirtual cart 420 does not include an error, generate receipt 1465 c.

c. Algorithm Feedback

In certain embodiments, virtual store tool 405 may be used inconjunction with algorithm 488, to improve the accuracy of thedeterminations made by algorithm 488. Specifically, machine learningmodule 480 may provide feedback to algorithm 488, based on a comparisonof the contents of virtual shopping cart 420 to the contents ofalgorithmic shopping cart 1420. FIG. 15 illustrates the operation ofmachine learning module 480.

As illustrated in FIG. 15, in certain embodiments, machine learningmodule 480 receives algorithmic shopping cart 1420 and virtual shoppingcart 420. Machine learning module 480 may then perform a comparison 1430of the contents of virtual shopping cart 420 and the contents ofalgorithmic shopping cart 1420, to determine if a discrepancy 1435exists between the two carts. In certain other embodiments, machinelearning module 480 may receive an indication that a discrepancy 1435exists between virtual shopping cart 420 and algorithmic shopping cart1420 directly from resolution component 475.

Discrepancy 1435 may include any inconsistency between virtual shoppingcart 420 and algorithmic shopping cart 1420. For example, discrepancy1435 may include an absence of a physical item 315 a from algorithmicshopping cart 1420, where the corresponding virtual item 320 a ispresent in virtual shopping cart 420. Such a discrepancy may occur whena weight sensor 1300 coupled to a physical shelf 305 on which physicalitem 315 a is placed, failed to detect the selection of the physicalitem 315 a from physical shelf 305. As another example, discrepancy 1435may include the presence of a physical item 315 b in algorithmicshopping cart 1420, where the corresponding virtual item 320 b is absentfrom virtual shopping cart 420. Such a discrepancy may arise fromalgorithm 488 failing to detect that a customer 105, who initiallyselected physical item 315 b from a physical rack 210, put item 315 bdown and did not leave physical store 100 with the item 315 b. As afurther example, discrepancy 1435 may include an identification swapbetween a first customer 105 a and a second customer 105 b, such that afirst item 315 a selected by first customer 105 a is absent from thealgorithmic shopping cart 1420 assigned to first customer 105 a, andpresent in an algorithmic shopping cart 1420 assigned to second customer105 b. Such an identification swap may occur in the customer trackingcomponent of algorithm 488.

In response to determining that a discrepancy exists between algorithmicshopping cart 1420 and virtual shopping cart 420, machine learningmodule 480 may determine a subset 1500 of inputs received by sensors 498(including cameras 1305 and weight sensors 1300) and associated with thediscrepancy. As an example, machine learning module 480 may determine atimestamp range of camera feed segments 415 and/or 425 during whichdiscrepancy 1435 occurred. For example, machine learning module 480 maydetermine that a virtual item 320 a was added to virtual shopping cart420, based on a portion of customer 105's shopping session capturedbetween a first timestamp and a second timestamp of camera feed segments415 and/or 425, but that a corresponding physical item 315 a was notadded to algorithmic shopping cart 1420. As a result, machine learningmodule 480 may determine a subset 1500 of inputs received from sensors498 during the same time interval (i.e., the time interval occurringbetween the first timestamp and the second timestamp). Subset 1500 mayinclude a subset 1505 of inputs received from cameras 1305 and/or asubset 1510 of inputs received from weight sensors 1300.

In response to identifying subset 1500, associated with discrepancy1435, machine learning module 480 may attach metadata 1540 to subset1500. This disclosure contemplates that metadata 1540 may include anyinformation explaining and/or addressing discrepancy 1435. For example,attaching metadata 1540 to subset 1500 may include attaching a label toone or more frames received from cameras 1305 indicating that the framesillustrate customer 105 a selecting item 315, rather than customer 105 bselecting the item, as originally determined by algorithm 488. Inresponse to attaching metadata 1540 to subset 1500, external system 485may use subset 1500 to retrain algorithm 488. In certain embodiments,retraining algorithm 488 may result in an improved accuracy of algorithm488.

Machine learning module 480 may be a software module stored in memory440 and executed by processor 435. An example of the operation ofmachine learning module 480 is as follows: (1) receive algorithmicshopping cart 1420; (2) receive virtual shopping cart 420; (3) comparethe contents of virtual shopping cart 420 to the contents of algorithmicshopping cart 1420; (4) determine that discrepancy 1435 exists betweenvirtual shopping cart 420 and algorithmic shopping cart 1420; (5)determine a subset 1500 of inputs received from sensors 498 (includingcameras 1305 and weight sensors 1300); (6) attach metadata 1540 tosubset 1500, so that external system 485 may use subset 1500 to retrainalgorithm 488.

FIG. 16 presents a flowchart illustrating the manner by which virtualstore tool 405 may use virtual shopping cart 420 to provide feedback toalgorithm 488. In step 1605, resolution component 475 receives analgorithmic shopping cart 1420. Algorithmic shopping cart 1420 includesa set of physical items 315, determined by algorithm 488 to have beenselected by a customer 105 during a shopping session in physical store100, based on inputs received from sensors 498 located in physical store100. In step 1610, resolution component 475 receives a virtual shoppingcart 420. Virtual shopping cart 420 includes a set of virtual items 320.In certain embodiments, virtual items 320 were determined by a user 120to have been selected by customer 105 during a shopping session inphysical store 100, based on camera feed segments 415 and/or 425capturing the shopping session of customer 105 in physical store 100.

In step 1615, resolution component 475 compares the contents ofalgorithmic shopping cart 1420 to the contents of virtual shopping cart420. In step 1620, resolution component 475 determines whether adiscrepancy 1435 exists between algorithmic shopping cart 1420 andvirtual shopping cart 420. If, in step 1620, resolution component 475determines that a discrepancy 1435 does not exist between algorithmicshopping cart 1420 and virtual shopping cart 420, then, in step 1640,resolution component 475 generates a receipt 1465 for the shoppingsession, based on the contents of virtual shopping cart 420, and sendsreceipt 1465 to a device 125 of customer 105. If, in step 1620,resolution component 475 determines that a discrepancy 1435 existsbetween algorithmic shopping cart 1420 and virtual shopping cart 420,then, in step 1625, machine learning module 480 determines a subset 1500of the set of inputs received from sensors 498 associated with thediscrepancy. In step 1630, machine learning module 480 attaches metadata1540 to subset 1500. Metadata 1540 may explain discrepancy 1435. In step1635, external system 485 may use subset 1500 to retrain algorithm 488.Additionally, in step 1640, resolution component 475 generates a receipt1465 for the shopping session, based on the contents of virtual shoppingcart 420, and sends receipt 1465 to a device 125 of customer 105.

Modifications, additions, or omissions may be made to method 1600depicted in FIG. 16. Method 1600 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as virtual store tool 405 (or componentsthereof) performing the steps, any suitable component of system 400,such as device(s) 115 for example, may perform one or more steps of themethod.

VII. Increasing the Efficiency of the Virtual Emulation Process

As described above, virtual store tool 405 may be used to virtuallyemulate a shopping session occurring in a physical store 100 andcaptured by camera feed segments 415 and/or 425. Using virtual storetool 405 in this manner may be a time-consuming process, especially ifthe tool is used to virtually emulate a large number of shoppingsessions. Accordingly, virtual shopping tool 405 includes a number offeatures to increase the efficiency of the virtual emulation process.These may include: (1) providing suggestions of items that customer 105may have selected during the shopping session, as determined by anexternal system 485; (2) enabling user 120 to efficiently navigatevideos of the shopping session displayed on graphical user interface700; and/or (3) providing an image of each customer 105 as he/she exitedthe physical store 100. Each of these features is described in furtherdetail in the discussion that follows.

a. Algorithmic Suggestions

As described above, in certain embodiments, virtual store tool 405 maybe used to independently verify determinations made by algorithm 488.For example, virtual store tool 405 may compare the contents of virtualshopping cart 420 to the contents of algorithmic cart 1420 to evaluatethe accuracy of algorithmic cart 1420 and/or to provide feedback toalgorithm 488, for use in training algorithm 488. In order to providesuch an independent verification, in certain embodiments, virtual storetool 405 does not receive any information about the contents ofalgorithmic cart 1420 until after virtual shopping cart 420 has beengenerated. This may be desirable to prevent any of the determinationsthat have been made by algorithm 488 from biasing user 120. Preventingthe determinations of algorithm 488 from biasing user 120 may beespecially important early on in the training process for algorithm 488,when there exists a reasonable probability that the determinations madeby algorithm 488 are incorrect, and/or when complex/unexpected scenariosarise within physical store 100, which may prove challenging foralgorithm 488 to accurately process.

On the other hand, in situations in which there exists a highprobability that the determinations made by algorithm 488 are correct(e.g., after algorithm 488 has undergone suitable training), the risk ofvirtual cart errors arising due to a biasing of user 120 towardserroneous determinations made by algorithm 488 may be minimal. At thesame time, efficiency gains may occur when user 120 uses virtual storetool 405 to generate virtual cart 420 by confirming suggestions providedby algorithm 488 of the items selected by customer 105 during a shoppingsession in physical store 100 (or a portion of a shopping session in aphysical store 100), rather than independently identifying the itemsselected by customer 105. For example, user 120 may be able to usevirtual store tool 405 to generate virtual cart 420 more rapidly whenaided by suggestions from algorithm 488.

i. Presenting, to a User of the Virtual Store Tool, a List of Productsthat have been Identified by an Algorithm as Selected During a ShoppingSession

FIGS. 17A and 17B present a first example of an embodiment in whichvirtual store tool 102 presents suggestions received from algorithm 488to user 120. Specifically, FIGS. 17A and 17B present an example of anembodiment in which virtual store tool 102 presents a list 1704 ofproducts that have been determined by algorithm 488 to have beenselected by a customer 105 a during a shopping session in physical store100 to a user 120 who is using virtual store tool 405 to virtuallyemulate the shopping session. User 120 may use the products in list 1704as suggestions/hints of the products user 120 should expect to viewcustomer 105 a selecting during his/her shopping session. FIG. 17Apresents an example of the components of system 400 that may be involvedin the process of virtual store tool 405 displaying product list 1704 onthe graphical user interface 700 that is presented to user 120 on device115, and FIG. 17B presents an example embodiment of graphical userinterface 700 that includes product list 1704.

As illustrated in FIG. 17A, in certain embodiments, virtual store tool405 receives an algorithmic shopping cart 1420 that includes a set ofphysical items 315 determined by an algorithm 488 of external system 485to have been selected by a customer 105 a during a shopping session inphysical store 100. Algorithm 488 may have determined that customer 105a selected the set of physical items 315 based on inputs received fromsensors 498 located in physical store 100. As described above, in thediscussion of FIGS. 13A through 13D, sensors 498 may include cameras1305 and weight sensors 1300 located in physical store 100. In responseto receiving algorithmic shopping cart 1420, in certain embodiments, foreach physical item 315 j/q included in algorithmic shopping cart 1420,virtual store tool 405 displays a corresponding virtual item 320 j/q ina product list 1704 on graphical user interface 700. For example, asillustrated in FIGS. 17A and 17B, product list 1704 includes virtualitem 320 j and virtual item 320 q, indicating that algorithm 488determined that customer 105 a selected physical item 315 j and physicalitem 315 q during a shopping session in physical store 100. In someembodiments, algorithmic shopping cart 1420 may be a partial shoppingcart corresponding to a portion of customer 105 a's shopping session inphysical store 100. For example, algorithmic shopping cart 1420 maycorrespond to a portion of customer 105 a's shopping session for whichthe inputs received from sensors 498 indicate that verification of thealgorithmic determination is desirable (e.g., the inputs indicate thatthere exists a probability that one or more items in algorithmicshopping cart 1420 is incorrect that is greater than a threshold).

A. Information Included in the Product List

Virtual store tool 405 may include any information received fromalgorithm 488 (e.g., algorithmic shopping cart 1420) in product list1704, and may display this information in any suitable manner. As anexample, while FIG. 17B depicts product list 1704 as a region ofgraphical user interface 700 in which virtual items 320 j/q aredisplayed as graphical representations of the corresponding physicalitems 315 j/q, virtual store tool 405 may present an identification ofitems 315 j/q to user 120 in any suitable manner. For example, incertain embodiments, product list 1704 includes textual descriptions ofphysical items 315 j/q determined by algorithm 488 to have been selectedby customer 105 a. As another example, product list 1704 may includeadditional information (other than an identification of physical items315 j/q determined by algorithm 488 to have been selected by customer105 a) that may be useful to user 120 in populating virtual shoppingcart 420. For example, in certain embodiments, product list 1704 mayindicate a quantity of each item 320 j/q that algorithm 488 determinedwas selected by customer 105 a during his/her shopping session. As anexample, product list 1704 may include a number to the left or right ofeach virtual item 320 j/q displayed within the list, indicating thequantity of the corresponding physical item 315 j/q that algorithm 488determined was selected by customer 105 a. In some embodiments, productlist 1704 does not include any indication of the quantity of eachphysical item 315 that algorithm 488 determined was selected by customer105 a. For example, in such embodiments, a product list 1704, asdepicted in FIG. 17B, may correspond to a situation in which algorithm488 has determined that customer 105 a selected a single physical item315 j and a single physical item 315 q (e.g., virtual store tool 405received an algorithmic shopping cart 1420 from external system 485 thatincluded a single physical item 315 j and a single physical item 315 q),as well as a situation in which algorithm 488 has determined thatcustomer 105 a selected 10 physical items 315 j and 2 physical items 315q (e.g., virtual store tool 405 received an algorithmic shopping cart1420 from external system 485 that include 10 physical items 315 j and 2physical items 315 q).

B. Timing of Displaying Items in the Product List

Virtual store tool 405 may add virtual products 320 j/q to product list1704 at any suitable point in time while user 120 is using graphicaluser interface 700 to virtually emulate a shopping session of a customer105 a. For example, virtual store tool 405 may display a virtual item320 j/q in product list 1704 for each physical item 315 j/q that isincluded in the algorithmic shopping cart 1420 that was received fromexternal system 485, at the start of the virtual emulation session. Forinstance, when graphical user interface 700 first displays product list1704 on display 410, product list 1704 may already include virtual items320 j/q. This may occur, for example, in embodiments in which customer105 a's shopping session (or the portion of customer 105 a's shoppingsession subject to virtual emulation) has already ended prior to the useof virtual store tool 405 to virtually emulate the shopping session (orthe portion of the shopping session subject to virtual emulation).virtual store tool Specifically, in certain embodiments, the length ofcamera feed segments 415 may correspond to and/or exceed the length ofcustomer 105 a's shopping session (or the length of the portion of theshopping session subject to virtual emulation), such that when virtualstore tool 405 displays camera feed segments 415 on graphical interface700, the shopping session (or portion of the shopping session) hasalready ended. In such embodiments, algorithm 488 may have already usedthe information collected from sensors 498 during the shopping session(or the portion of the shopping session subject to virtual emulation) topopulate algorithmic shopping cart 1420, and subsequently provided thecart to virtual store tool 405 prior to virtual store tool 405displaying graphical user interface 700. Accordingly, virtual store tool405 may display a virtual item 320 j/q in product list 1704 for eachphysical item 315 j/q included in algorithmic cart 1420, when virtualstore tool 405 first displays graphical user interface 700 at thebeginning of the virtual emulation session.

Product list 1704 may also include one or more virtual items 320 j whenfirst displayed on graphical user interface 700 in embodiments in whichvirtual store tool 405 is being used to virtually emulate a shoppingsession (or a portion of a shopping session) that is in progress.Specifically, in certain embodiments, the length of camera feed segments415 is less than the length of customer 105 a's shopping session (of theportion of the shopping session subject to virtual emulation), such thatwhen virtual store tool 405 initially displays camera feed segments 415on graphical interface 700, the shopping session is still ongoing. Insuch embodiments, algorithm 488 may have already populated algorithmicshopping cart 1420 with one or more physical items 315 j that itdetermined customer 105 a selected during a first portion of theshopping session captured by camera feed segments 415. Algorithm 488 mayprovide this partial algorithmic shopping cart 1420 to virtual storetool 405, enabling the tool to display a virtual item 320 j in productlist 1704 for each physical item 315 j included in partial algorithmiccart 1420. In such embodiments, virtual store tool 405 may updateproduct list 1704 in response to algorithm 488 adding additionalphysical items 315 q to algorithmic shopping cart 1420. For example, incertain embodiments, virtual store tool 405 may update product list 1704in conjunction with displaying a new set of camera feed segments 415,which correspond to a time interval immediately following the timeinterval associated with the previous set of camera feed segments 415.The updated product list 1704 may include virtual items 320 qcorresponding to physical items 315 q determined by algorithm 488 tohave been selected by customer 105 during the time interval covered bythe new camera feed segments 415. In some embodiments, virtual storetool 405 may update product list 1704 in response to receiving anupdated algorithmic shopping cart 1420 that includes a new physical item315 q.

As another example of the timing at which virtual store tool 405 maydisplay virtual items 320 j/q in product list 1704, in certainembodiments, when virtual store tool 405 first displays graphical userinterface 700, product list 1704 is empty. In such embodiments, virtualstore tool 405 may add virtual items 320 j/q to product list 1704 as thevirtual emulation session progresses (e.g., as the playback of camerafeed segments 415, which captured the shopping session, progresses). Asa specific example, in certain embodiments, virtual store tool 405 addsa virtual item 320 j (corresponding to a physical item 315 j) to productlist 1704 when the playback progress of camera feed segments 415 and/or425 reaches a specified time interval before the playback time at whichcamera feed segments 415 and/or 425 should have captured customer 105 aselecting physical item 315 j, as determined by algorithm 488. Forexample, consider a scenario in which the specified time interval is 15seconds, and algorithm 488 has determined that customer 105 a selectedphysical item 315 j at 3:15:10 pm. In such embodiments, virtual carttool 405 will then add virtual item 320 j to product list 1704 when theplayback progress of camera feed segments 415 and/or 425 reaches aplayback time corresponding to 3:14:55 pm.

C. Use of the Product List

The virtual items 320 j/q displayed in product list 1704 may aid user120 in efficiently emulating a shopping session of a customer 105 a inphysical store 100. In particular, because the virtual items 320 j/qdisplayed in product list 1704 correspond to physical items 315 j/qdetermined by algorithm 488 to have been selected by customer 105 aduring a shopping session in physical store 100, the virtual items 320j/q displayed in product list 1704 may provide suggestions to user 120of items 320 j/q to consider adding to virtual shopping cart 420. User120 may use the suggestions provided by product list 1704 in anysuitable manner.

As a first example, in certain embodiments, user 120 may select forconsideration one of the virtual items 320 j/q displayed in product list1704 and determine whether customer 105 a selected the correspondingphysical item 315 j/q during his/her shopping session in physical store100. User 120 may select for consideration a virtual item 320 j/q fromproduct list 1704 in any suitable manner. For example, user 120 mayselect virtual item 320 j for consideration by clicking on the graphicalicon corresponding to the virtual item, double-clicking on the graphicalicon, and/or interacting with the graphical icon in any other suitablemanner.

Graphical user interface 700 may include any of a number of featuresthat may help user 120 in directing his/her focus towards a givenvirtual item 320 j/q that is displayed in product list 1704 and that hasbeen selected for consideration by user 120. As an example, in responseto selecting a first virtual item 320 j displayed in product list 1704,virtual store tool 405 may highlight, in virtual layout 205, the virtualrack 230 m on which virtual item 320 j is stored. Highlighting virtualrack 230 m may include any method of distinguishing virtual rack 230 mfrom the other virtual racks 230. For example, highlighting virtual rack230 m may include placing a frame around virtual rack 230 m, applying acolor/shading to virtual rack 230 m (as illustrated in FIG. 17B), and/orany other suitable method of distinguishing virtual rack 230 m from theremaining virtual racks 230. As another example, in certain embodiments,in response to selecting a first virtual item 320 j displayed in productlist 1704, virtual store tool 405 may display, in region 905 ofgraphical user interface 700, the virtual rack 230 m on which virtualitem 320 j is stored. In some such embodiments, virtual store tool 405may also highlight virtual item 320 j on virtual rack 230 m.Highlighting virtual item 320 a may include any method of distinguishingvirtual item 320 j from the remaining virtual items 320 on virtual rack230 m. For example, highlighting virtual item 320 j may include placinga frame around virtual item 320 j, enlarging virtual item 320 j comparedto the other virtual items 320, changing a color of the backgroundaround virtual item 320 j, and/or any other suitable method ofdistinguishing virtual item 320 j from the remaining virtual items 320.As a further example, in certain embodiments, in response to selectingvirtual item 320 j displayed in product list 1704, virtual store tool405 may display camera feed segment 1702, which includes video of thephysical rack 210 m in physical store 100 to which virtual rack 230 m isassigned. For example, as illustrated in FIG. 17A, camera feed segment1702 may have been captured by a rack camera 495 directed at thephysical rack 210 m to which virtual rack 230 m is assigned. Asillustrated in FIG. 17B, in certain embodiments, virtual store tool 405displays camera feed segment 1702 on graphical user interface 700 in aseparate region of display 410 from camera feed segments 415 a through415 f, and/or enlarged as compared to camera feed segments 415 a through415 f. This may aid user 120 in directing his/her attention towardscamera feed segment 1702.

As a second example of the manner by which user 120 may use thesuggestions provided by product list 1704, in certain embodiments, user120 may add any of the virtual items 320 j/q displayed in product list1704 to virtual shopping cart 420, while virtually emulating a shoppingsession of a customer 105 a in physical store 100. For example, user 120may consider a first virtual item 320 j displayed in product list 1704.As described above, by displaying first virtual item 320 j in productlist 1704, virtual store tool 405 may indicate to user 120 that anexternal system 485 has determined that customer 105 a selected thecorresponding physical item 315 j during the shopping session. User 120may then confirm that customer 105 a selected the corresponding physicalitem 315 j by observing customer 105 a selecting physical item 315 j oncamera feed segment 1702 and/or any of camera feed segments 415 athrough 415 f, displayed on graphical user interface 700. In response toconfirming that customer 105 a selected physical item 315 j, user 120may add virtual item 320 j to virtual shopping cart 420, therebyvirtually emulating customer 105 a's selecting of physical item 315 j.In certain embodiments, user 120 may add virtual item 320 j to virtualshopping cart 420 by selecting virtual item 320 j directly from productlist 1704. For example, user 120 may (1) drag virtual item 320 j fromproduct list 1704 to virtual shopping cart 420 and drop virtual item 320j in virtual shopping cart 420, (2) double-click on virtual item 320 jin product list 1704 to add virtual item 320 j to virtual shopping cart420, and/or (3) interact with graphical user interface 700 in any othersuitable way to indicate that virtual item 320 j should be moved fromproduct list 1704 to virtual shopping cart 420. In certain embodiments,user 120 may also or alternatively add virtual item 320 j to virtualshopping cart 420 by selecting virtual item 320 j from virtual rack 230m, as described in Part IV above. In certain embodiments, in response touser 120 placing virtual item 320 j in virtual shopping cart 420,virtual store tool 405 removes virtual item 320 j from product list1704.

As a third example of the manner by which user 120 may use thesuggestions provided by product list 1704, in certain embodiments, user120 may use product list 1704 to identify errors in algorithmic shoppingcart 1420. For example, user 120 may consider a second virtual item 320q, corresponding to physical item 315 q, presented in product list 1704.User 120 may then observe camera feed segment 1702 and/or any of camerafeed segments 415 a through 415 f that include video of physical rack210 m, on which physical item 315 q is stored, to determine thatcustomer 105 a did not select physical item 315 q during his/hershopping session in physical store 100. Accordingly, because the virtualitems 320 j/q displayed in product list 1704 correspond to physicalitems 315 j/q that algorithm 488 determined were selected by customer105 a during the shopping session, by determining that customer 105 adid not select physical item 315 q, user 120 determined that theinclusion of physical item 315 q in algorithmic shopping cart 1420 is anerror. In certain embodiments, in response to determining that customer105 a did not select physical item 315 q, user 120 may instruct virtualstore tool 405 to remove virtual item 320 q from product list 1704. Insome embodiments, the removal of virtual item 320 q from product list1704 and/or the lack of inclusion of virtual item 320 q in virtualshopping cart 420 may be used to provide feedback to algorithm 488, in asimilar manner as described above, in the discussion of FIGS. 15 and 16.

ii. Prepopulating a Virtual Shopping Cart with Products that have beenIdentified by an Algorithm as Selected During a Shopping Session, Priorto Virtual Emulation of that Shopping Session

In certain embodiments, virtual store tool 102 may present suggestionsreceived from algorithm 488 to user 120 by placing these suggestionsdirectly in virtual shopping cart 420. For example, when graphical userinterface 700 first displays virtual shopping cart 420 on display 410,virtual shopping cart 420 may already include virtual items 320 j/q,where each virtual item 320 j/q corresponds to a physical item 315 j/qdetermined by algorithm 488 to have been selected by customer 105 aduring his/her shopping session in physical store 100. User 120 may thenview camera feed segments 1702 and/or 415 a through 415 f to determinewhether the prepopulated virtual shopping cart 420 accurately reflectsthe physical items 315 j selected by customer 105 a during his/hershopping session. If user 120 determines that the prepopulated virtualshopping cart 420 does not accurately reflect the physical items 315 jselected by customer 105 a during his/her shopping session, user 120 maymodify virtual shopping cart 420 by: (1) adjusting a quantity of avirtual item 320 j in virtual shopping cart 420; (2) removing a virtualitem 320 q from virtual shopping cart 420; and/or (3) adding a virtualitem 320 to virtual shopping cart 420.

In a similar manner as described above for embodiments in which thealgorithmic suggestions are displayed in product list 1704, in certainembodiments in which virtual shopping cart 420 is prepopulated withalgorithmic suggestions of virtual items 320 j/q, user 120 may interactwith the graphical icons assigned to the virtual items 320 j/q that aredisplayed in the prepopulated virtual shopping cart 420. In particular,user 120 may interact with a graphical icon assigned to a virtual item320 j/q to cause virtual store tool 405 to display information that mayaid user 120 in determining whether customer 105 a actually selected thecorresponding physical item 315 j/q during his/her shopping session inphysical store 100, thereby evaluating the algorithmic suggestion ofvirtual item 302 j/q. For example, in certain embodiments, in responseto selecting a first virtual item 320 j displayed in virtual shoppingcart 420 (e.g., clicking on the graphical icon corresponding to thevirtual item, double-clicking on the graphical icon, and/or interactingwith the graphical icon in any other suitable manner), virtual storetool 405 may: (1) highlight, in virtual layout 205, the virtual rack 230m on which virtual item 320 j is stored; (2) display, in region 905 ofgraphical user interface 700, the virtual rack 230 m on which virtualitem 320 j is stored; (3) highlight, on virtual rack 230 m, virtual item320 j; (4) display camera feed segment 1702, which includes video of thephysical rack 210 m of physical store 100 to which virtual rack 230 m isassigned; and/or (5) display any other suitable information that may aiduser 120 in evaluating the accuracy of the contents of prepopulatedvirtual shopping cart 420. As described above, highlighting virtual rack230 m in virtual layout 205 may include placing a frame around virtualrack 230 m, applying a color/shading to virtual rack 230 m, and/or anyother suitable method of distinguishing virtual rack 230 m from theremaining virtual racks 230. Similarly, as described above, highlightingvirtual item 320 j on virtual rack 230 m may include placing a framearound virtual item 320 j, enlarging virtual item 320 j compared to theother virtual items 320, changing a color of the background aroundvirtual item 320 j, and/or any other suitable method of distinguishingvirtual item 320 j from the remaining virtual items 320.

iii. Using Video Markers and Highlighting to Present AlgorithmicSuggestions to a User

In addition to/instead of presenting algorithmic suggestions to user 120as product list 1704 and/or in virtual shopping cart 420, virtual storetool 405 may present algorithmic suggestions to user 120 in severaldifferent manners, some of which were presented in Section IV above. Forexample, as discussed in Section IV, in certain embodiments, virtualstore tool 102 presents suggestions received from algorithm 488 to user120 by displaying markers 715 a/b on slider bar 705, where each marker715 a/b indicates that an event occurred within physical store 100 atthe marker playback time. Such events may include customer 105 aapproaching a physical rack 210 a and/or selecting an item 305 fromphysical rack 210 a, where the occurrence of the event was detected byalgorithm 488, based on inputs received from sensors 498 located inphysical store 100. As another example, as discussed in Section IVabove, in certain embodiments, virtual store tool 102 presentssuggestions received from algorithm 488 to user 120 by highlighting avirtual rack 230 in virtual layout 205, and/or highlighting a virtualitem 320 on virtual rack 230. For example, virtual store tool 102 mayhighlight virtual rack 230 a in response to receiving an indication fromalgorithm 488 that an event associated with physical rack 210 a occurred(e.g., customer 105 a approaching physical rack 210 a and/or selectingan item 305 from physical rack 210 a). Similarly, virtual store tool 102may highlight virtual item 320 f on virtual rack 230 a in response toreceiving an indication from algorithm 488 that customer 105 a selectedphysical item 315 f from physical rack 210 a.

Virtual store tool 405 may also apply highlighting to virtual items 320j/q that are displayed on virtual rack 230 m to distinguish betweenthose items that are displayed on the rack and are also stored inalgorithmic shopping cart 1420 and those items that are displayed on therack and are also stored in virtual shopping cart 420. As an example, incertain embodiments, virtual store tool 405 may display a first type ofhighlighting around each virtual item 320 j/q displayed on virtual rack230 m (displayed in region 905 of graphical user interface 700) thatcorresponds to a physical item 315 j/q that algorithm 488 determinedcustomer 105 selected during a shopping session in physical store 100.Virtual store tool 405 may also display a second type of highlightingaround each virtual item 320 j (displayed on virtual rack 230 m inregion 905 of graphical user interface 700) that is also present invirtual shopping cart 410. Displaying the first and second types ofhighlighting around virtual items 320 j/q may include placing framesaround virtual items 320 j/q, changing the color of the backgroundsurrounding virtual items 320 j/q, and/or any other manner ofdistinguishing virtual items 320 j/q from the remaining virtual itemsdisplayed on virtual rack 230 m in region 905 of graphical userinterface 700. In certain embodiments, the first type of highlighting isdifferent from the second type of highlighting. For example, the firsttype of highlighting may correspond to a first color frame and/orbackground around a virtual item 320 j, and the second type ofhighlighting may correspond to a second color frame and/or backgroundaround a virtual item 320 q. As a specific example of the use of thesetwo types of highlighting, virtual store tool 405 may initially changethe background around virtual items 320 j and 320 q (displayed onvirtual rack 230 in region 905) to a first color (e.g., red), inresponse to receiving information indicating that the correspondingphysical items 315 j and 315 q were added by algorithm 488 toalgorithmic shopping cart 1420. Then, in response to user 120 addingvirtual item 320 j to virtual shopping cart 420, virtual store tool 405may change the background around virtual item 320 j from the first colorto a second color (e.g., green). In this manner, when user 120 viewsvirtual rack 230 m (as displayed in region 905 of graphical userinterface 700) he/she may easily see, based on the presence of thesecond type of highlighting around virtual item 320 j, that he/she hasalready considered virtual item 320 j and determined that customer 105 aselected the corresponding physical item 315 j (e.g., user 120 hasalready added virtual item 320 j to virtual shopping cart 420).Similarly, based on the presence of the first type of highlightingaround virtual item 320 q, user 120 may easily determine that he/she hasnot yet considered the algorithmic suggestion of virtual item 320 q(and/or that he/she may have already considered the algorithmicsuggestion but determined that customer 105 a did not select thecorresponding physical item 315 q). Accordingly, user 120 may directhis/her attention towards virtual item 320 q in order to evaluate thisalgorithmic suggestion (and/or to confirm a previous determination thatthe algorithmic suggestion was incorrect).

Placing highlighting around a virtual item 320 j may also include addingany suitable information to the highlighting. For example, placing thefirst type of highlighting around a virtual item 320 j may includeplacing an indication of the quantity of the corresponding physical item315 j that is present in algorithmic shopping cart 1420. Similarly,placing the second type of highlighting around a virtual item 320 q mayinclude placing an indication of the quantity of the virtual item invirtual shopping cart 420. As a specific example of the use of these twotypes of highlighting, virtual store tool 405 may initially place framesof a first color (e.g., red) around virtual items 320 j and 320 q(displayed on virtual rack 230 in region 905), in response to receivinginformation indicating that the corresponding physical items 315 j and315 q were added by algorithm 488 to algorithmic shopping cart 1420.Virtual store tool 405 may also indicate within each frame a quantity ofthe corresponding physical item 315 j/q that is present in algorithmicshopping cart 1420. Then, in response to user 120 adding a differentquantity of virtual item 320 j to virtual shopping cart 420, virtualstore tool 405 may change the color of the frame around virtual item 320j from the first color to a second color (e.g., green), and may updatethe quantity indicated within the frame to correspond to the quantity ofvirtual item 320 j included in virtual shopping cart 420. The use ofhighlighting in this manner may help user 120 to easily identify virtualitems 320 j to which the user may wish to direct further attention. Forexample, prior to issuing a receipt to customer 105 a, based on virtualshopping cart 420, user 120 may wish to double-check one or more of thedecisions that he/she made while populating virtual shopping cart 420that conflicted with determinations made by algorithm 488. Accordingly,user 120 may use the second type of highlighting to easily identifythose virtual items 320 j for which the user's decisions conflicted withthe algorithm's determinations.

Virtual store tool 405 may additionally apply highlighting to physicalitems 315 j/q that are depicted in camera feed segments 415/1702. As anexample, in certain embodiments, virtual store stool 405 may place abounding box around each physical item 315 j/q depicted in camera feedsegments 415/1702. This bounding box may correspond to a bounding boxthat was used by algorithm 488 to track items selected by customer 105 ain physical store 100. In certain embodiments, the bounding box around aphysical item 315 j/q is displayed in camera feed segments 415/1702using the same color as the highlighting applied to the correspondingvirtual item 320 j/q.

b. Video Display and Control

While providing algorithmic suggestions to user 120 is one way in whichvirtual store tool 405 may aid user 120 in efficiently emulating ashopping session of customer 105, graphical user interface 700 mayinclude a number of additional features to help increase efficiency. Asan example, in certain embodiments, virtual store tool 405 enables auser 120 to interact with camera feed segments 415/1702 to zoom in on anarea of interest and/or pan over to an area of interest. As anotherexample, in certain embodiments, virtual store tool 405 displays atleast one camera feed segment 1702 in a somewhat central location ongraphical user interface 700, separate from the remaining camera feedsegments 415 a through 415 f. In some such embodiments, camera feedsegment 1702 is enlarged as compared with the remaining camera feedsegments 415 a through 415 f. For example, as illustrated in FIG. 17,virtual store tool 405 may place enlarged camera feed segment 1702 nearthe upper center of graphical user interface 700. Camera feed segment1702 may correspond to the camera feed segment currently of interest touser 120. Displaying camera feed segment 1702 separate from and/orenlarged as compared to camera feed segments 415 may aid user 120 inmaintaining his/her focus on the camera feed segment.

Enlarged camera feed segment 1702 may correspond to any of the camerafeeds received by virtual store tool 405 from layout cameras 490 and/orrack cameras 495. As an example, in certain embodiments, and asillustrated in FIG. 17B, camera feed segment 1702 may correspond to oneof the camera feed segments 415 a through 415 f (e.g., camera feedsegment 415 c), which may also be displayed on graphical user interface700, for example, to the right of camera feed segment 1702. In certainembodiments, user 120 may select one of camera feed segments 415 athrough 415 f for display as camera feed segment 1702. For example,virtual store tool 405 may display camera feed segment 415 c as camerafeed segment 1702, in response to user 120 selecting camera feed segment415 c (e.g., clicking on camera feed segment 415 c, double-clicking oncamera feed segment 415 c, and/or any other suitable manner of selectingcamera feed segment 415 c from the set of camera feed segments 415 athrough 415 f, displayed on graphical user interface 700).

As another example, in certain embodiments, camera feed segment 1702 maycorrespond to a camera feed segment received from a rack camera 495located in physical store 100. For example, in certain embodiments,virtual store tool 405 may display a camera feed segment received from arack camera 495 directed at a physical rack 210 m on which physical item315 j is stored as camera feed segment 1702, in response to user 120selecting a virtual item 320 j (configured to emulate physical item 315j) from product list 1704. In some embodiments, virtual store tool 405may display a camera feed segment received from a rack camera 495directed at a physical rack 210 m as camera feed segment 1702, inresponse to user 120 selecting virtual rack 230 m (configured to emulatephysical rack 210 m) from virtual layout 205. In certain embodimentsmultiple rack cameras 495 may be assigned to a given physical rack 210m. In such embodiments, virtual store tool 405 may display icons 1706 onvirtual layout 205, where the locations of icons 1706 on virtual layout205 emulate the physical locations of rack cameras 495 within physicallayout 200. In response to a user 120 selecting an icon 1706 a/b (e.g.,clicking on the icon, double-clicking on the icon, and/or any othersuitable method of selecting the icon), virtual store tool 405 maydisplay, as camera feed segment 1702, a camera feed segment that wascaptured by the rack camera 495 corresponding to the icon 1706 a/b.

c. Images from Entrance and Exit Cameras

As another example of a way in which virtual store tool 405 may aid user120 in efficiently emulating a shopping session of customer 105 a inphysical store 100, in certain embodiments, virtual store tool 405displays an image 1710 of customer 105 a, captured when the customerexited physical store 100. For example, in certain embodiments in whichphysical store 100 includes turnstiles 510 to control the entry/exit ofpersons 105 into/out of the store (as depicted in FIG. 5A), physicalstore 100 may include a camera configured to take an image 1710 ofcustomer 105 a as customer 105 a exits through a turnstile 510.Displaying image 1710 may help user 120 to easily confirm the accuracyof virtual shopping cart 420. For example, user 120 may use image 1710to confirm that virtual shopping cart 420 is accurate, where virtualshopping cart 420 includes a single virtual item 320 j and image 1710depicts customer 105 a exiting physical store 100 while carrying thecorresponding physical item 315 j. Similarly, user 120 may use image1710 to determine that virtual shopping cart 420 is inaccurate, wherevirtual shopping cart 410 includes a single virtual item 320 j and image1710 depicts customer 105 a exiting physical store 100 without thecorresponding physical item 315 j and/or with a different physical item315 a.

In addition to or instead of displaying image 1710 of customer 105 a, incertain embodiments, and as described above in the discussion of FIG.7B, virtual store tool 405 displays an image 725 of customer 105 a,captured when the customer entered physical store 100. Displaying image725 may help user 120 to easily identify the customer 105 a whoseshopping session virtual store tool 405 is being used to virtuallyemulate. This may be desirable, for example, in embodiments in whichmultiple customers are depicted in video segments 415 a through 415 fand/or 1702. As a specific example, in certain embodiments in whichvirtual store tool 102 presents a list 1704 of products that have beendetermined by algorithm 488 to have been selected by a customer 105 aduring a shopping session in physical store 100, algorithm 488 may havemistakenly tracked the wrong customer (i.e., a customer other thancustomer 105 a). In such embodiments, video segments 415 a through 415 fand/or 1702 may indeed depict a customer selecting one or more of theitems displayed in product list 1704; however, this customer is not thecustomer 105 a to which algorithm 488 assigned the items. Accordingly,by enabling user 120 to identify the customer 105 a for whom the virtualemulation is being performed, image 725 and/or 1710 may help user 120 todetermine that customer 105 a did not select the one or more itemsdisplayed in product list 1704 that were selected by the other customer.

d. Method for Virtually Emulating a Physical Shopping Session UsingAlgorithmic Suggestions

FIG. 18 presents a flowchart (described in conjunction with elements ofFIGS. 17A and 17B) illustrating an example method 1800 by which virtualstore tool 405 may present algorithmic suggestions to user 120, and user120 may use these suggestions while virtually emulating a shoppingsession of a customer 105 a in physical store 100. Specifically, method1800 illustrates the use of algorithmic suggestions in the form ofproduct list 1704, where each virtual item 320 displayed in product list1704 corresponds to a physical item 315 that algorithm 488 determinedthat customer 105 a selected during the shopping session.

In step 1802 virtual store tool 405 receives algorithmic shopping cart1420. For example, virtual store tool 405 may receive algorithmicshopping cart 1420 from external system 485. Algorithmic shopping cart1420 may include a set of physical items 315 determined by algorithm 488to have been selected by customer 105 a, based on inputs received fromsensors 498 located in physical store 100. In step 1804 virtual storetool 405 uses algorithmic shopping cart 1420 to display product list1704. For example, for each physical item 315 present in algorithmicshopping cart 1420, virtual store tool 405 displays a correspondingvirtual item 320 in product list 1704. In certain embodiments, productlist 1704 includes an indication of the quantity of each physical item315, correspond to a virtual item 320 displayed in product list 1704,that algorithm 488 determined was selected by customer 105. For example,in certain embodiments, product list 1704 may include a number to theleft or right of each virtual item 320 displayed within the list,indicating the quantity of the corresponding physical item 315 thatalgorithm 488 determined was selected by customer 105 a.

In step 1806 virtual store tool 405 considers a first virtual item 320from product list 1704. For example, in certain embodiments, virtualstore tool 405 receives an indication that user 120 has selected thevirtual item 320 from product list 1704. User 120 may have selectedvirtual item 320 from product list 1704 in any suitable manner. Forexample, in certain embodiments, selecting virtual item 320 from productlist 1704 may include clicking on a graphical icon displayed in productlist 1704 that corresponds to virtual item 320, double-clicking on thegraphical icon, and/or interacting with the graphical icon in any othersuitable manner. In step 1808 virtual store tool 405 displays videosegment 1702. Video segment 1702 includes video of the physical rack 210in physical store 100 on which the physical item 315 corresponding tothe selected virtual item 320 resides. In certain embodiments, videosegment 1702 is a camera feed segment captured by a rack camera 495 thatwas directed at the physical rack 210 on which the physical item 315corresponding to the selected virtual item 320 resides.

In step 1810 virtual store tool 405 determines whether the tool hasreceived information identifying the virtual item 320 selected fromproduct list 1704. This information may include any information that mayindicate to virtual store tool 405 that the tool should store theselected virtual item 320 in virtual shopping cart 420. As an example,in certain embodiments, receiving information identifying the selectedvirtual item 320 includes receiving information associated with draggingand dropping virtual item 320 from product list 1704 to virtual shoppingcart 420. As another example, in certain embodiments, receivinginformation identifying the selected virtual item 320 includes receivinginformation associated with dragging and dropping virtual item 320 fromvirtual rack 230, displayed in region 905 of display 410, to virtualshopping cart 420.

If, in step 1810 virtual store tool 405 determines that the tool hasreceived information identifying the selected virtual item 320, in step1820 virtual store tool 405 adds virtual item 320 to virtual shoppingcart 420. Method 1800 then proceeds to step 1822. If, in step 1810virtual store tool 405 determines that the tool has not receivedinformation identifying the selected virtual item 320, in step 1812, incertain embodiments, virtual store tool 405 removes virtual item 320from product list 1704. As an example, in certain embodiments, virtualstore tool 405 receives an indication that customer 105 a did not selectthe physical item 315 corresponding to virtual item 320 during theshopping session. For instance, virtual store tool 405 may receive anindication from user 120 that user 120 did not view customer 105 aselecting the physical item 315 corresponding to virtual item 320 incamera segments 415/1702. In response to receiving the indication thatcustomer 105 a did not select the physical item 315 corresponding tovirtual item 320 during the shopping session, virtual store tool 405removes virtual item 320 from product list 1704. As another example, incertain embodiments, virtual store tool 405 removes virtual item 320from product list 1704 if virtual store tool 405 does not receiveinformation identifying virtual item 320 before: (1) playback of videosegment 1702 completes, (2) virtual store tool 405 considers a nextvirtual item 320 from product list 1704, and/or (3) virtual store tool405 issues a receipt for the shopping session.

In step 1814 virtual store tool 405 determines whether the tool receivedinformation identifying a virtual item 320 that is not included inproduct list 1704. As an example, in certain embodiments, receivinginformation identifying a virtual item 320 that is not included inproduct list 1704 includes receiving information associated withdragging and dropping the virtual item 320 from a virtual rack 230,displayed in region 905 of display 410, to virtual shopping cart 420.For example, user 120 may drag and drop the virtual item 320 fromvirtual rack 230 to virtual shopping cart 420 in response to viewing, incamera segment 1702, customer 105 a selecting the virtual item 320. If,in step 1814 virtual store tool 405 determines that the tool receivedinformation identifying a virtual item 320 that is not included inproduct list 1704, in step 1816 virtual store tool 405 adds the virtualitem 320 to virtual shopping cart 420. Method 1800 then proceeds to step1822. If, in step 1814 virtual store tool 405 determines that the tooldid not receive information identifying a virtual item 320 that is notincluded in product list 1704, method 1800 proceeds directly to step1822.

In step 1822 virtual store tool 405 determines whether any additionalvirtual items 320 are present in product list 1704. If, in step 1822virtual store tool 405 determines that one or more additional virtualitems 320 are present in product list 1704, method 1800 returns to step1806. If, in step 1822 virtual store tool 405 determines that noadditional virtual items 320 are present in product list 1704, in step1824 virtual store tool 405 determines whether the tool receivedinformation identifying a virtual item 320 that is not included inproduct list 1704. As an example, in certain embodiments, receivinginformation identifying a virtual item 320 that is not included inproduct list 1704 includes receiving information associated withdragging and dropping the virtual item 320 from a virtual rack 230,displayed in region 905 of display 410, to virtual shopping cart 420.For example, user 120 may drag and drop the virtual item 320 fromvirtual rack 230 to virtual shopping cart 420 in response to viewing, incamera segment 1702, customer 105 a selecting the virtual item 320. If,in step 1824 virtual store tool 405 determines that the tool receivedinformation identifying a virtual item 320 that is not included inproduct list 1704, in step 1826 virtual store tool 405 adds the virtualitem 320 to virtual shopping cart 420. Method 1800 then proceeds to step1828. If, in step 1824 virtual store tool 405 determines that the tooldid not receive information identifying a virtual item 320 that is notincluded in product list 1704, method 1800 proceeds directly to step1828. In step 1828 virtual store tool 405 determines whether theshopping session has ended. If, in step 1828 virtual store tool 405determines that the shopping session has not ended, method 1800 returnsto step 1824. If, in step 1828 virtual store tool 405 determines thatthe shopping session has ended, method 1800 ends.

Modifications, additions, or omissions may be made to method 1800depicted in FIG. 18. Method 1800 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as virtual store tool 405 (or componentsthereof) performing the steps, any suitable component of system 400,such as device(s) 115 for example, may perform one or more steps of themethod.

VIII. Evaluation of a Refund Request through Virtual Emulation of aPrior Shopping Session

As described above, a customer 105 may be charged for items that he/sheselected during a shopping session in a physical store 100 based on: (1)a virtual emulation of the shopping session, using virtual store tool405; (2) an algorithmic determination, based on information gathered bysensors 498 in the physical store during the shopping session, of itemsselected by the customer; or (3) a combination of virtual emulation andalgorithmic determinations. Regardless of the manner by which thecharges are determined, in certain embodiments, customer 105 may receivea receipt 1465 detailing the charges that were applied to his/heraccount. Customer 105 may dispute one or more of these charges, for avariety of reasons. As an example, in certain embodiments where virtualstore tool 405 was used to generate the charges, an error may haveoccurred in the virtual emulation process such that one or more of theitems for which customer 105 was charged were not actually selected bythe customer during his/her shopping session and/or customer 105 did notselect the quantity of one of more of the items for which customer 105was charged. As another example, in certain embodiments wherealgorithmic shopping cart 1420 was used to generate the charges,customer 105 may not have actually selected one or more of the itemsthat algorithm 488 determined customer 105 selected, and/or a quantitythat algorithm 488 assigned to one or more of the items included inalgorithmic shopping cart 1420 may not be accurate. As a furtherexample, in certain embodiments, the charges may be accurate butcustomer 105 may have misremembered the items that he/she selectedduring his/her shopping session.

FIGS. 19A through 19C present an example illustrating the use of virtualstore tool 405 to process a refund request 1902 a submitted by acustomer 105 a. As illustrated in FIG. 19A, refund request 1902 a is arequest for a refund of the purchase price of physical item 315 w thatwas charged to an account belonging to customer 105 a in response to aprior shopping session in physical store 100. While illustrated in FIG.19A as corresponding to a request for the refund of the purchase priceof a single physical item 315 w, virtual store tool 405 may be used toprocess requests for refunds of the purchase price of any number ofphysical items.

a. Receiving a Refund Request

Customer 105 a may submit refund request 1902 a to virtual store tool405 in any suitable manner. As an example, in certain embodiments,customer 105 a submits refund request 1902 a to virtual store tool 405through a webpage. As another example, in certain embodiments, customer105 a submits refund request 1902 a to virtual store tool 405 usinghis/her mobile device 125 a. For example, in certain embodiments,receipt 1465 a includes an interactive link through which customer 105 amay submit refund request 1902 a. In certain such embodiments, inresponse to receiving receipt 1465 a on device 125 a, customer 105 auses device 125 a to interact with the link to submit refund request1902 a. In some embodiments, customer 105 a may use an applicationinstalled on device 125 a to submit refund request 1465 a.

In response to receiving refund request 1465 a, virtual store tool 405places refund request 1465 a in queue 1906, stored in memory 440.Virtual store tool 405 uses queue 1906 to help ensure that the refundrequests it receives are processed in a timely manner. In particular, asillustrated in FIG. 19B, any number of customers 105 a through 105 n maysubmit refund requests 1902 a through 1902 o to virtual store tool 405.In certain embodiments, there may not be any users 120 a through 120 mwho are available to process the incoming refund requests 1902 a through1902 o as they are received. For example, users 120 a through 120 m maynot be available to process an incoming refund request 1902 a becausethey are virtually emulating shopping sessions and/or processingearlier-received refund requests. Virtual store tool 405 uses queue 1906to help ensure that when a user 120 becomes available, he/she isprovided with the earliest-received refund request that has not yet beenprocessed, thereby helping to ensure that refund requests 1902 a through1902 o are processed in the order that they are received by virtualstore tool 405.

As an example of the manner by which virtual store tool 405 uses queue1906 to assign refund requests 1902 a through 1902 o to users 120 athrough 120 m, as illustrated in FIG. 19B, a first refund request 1902a, submitted by customer 105 a, is stored in a first slot 1908 a ofqueue 1906, a second refund request 1902 c is stored in a second slot1908 b of queue 1906, and an n-th refund request 1902 n is stored in ann-th slot 1908 n of queue 1906. In response to receiving refund request1902 o from customer 105 b, virtual store tool 405 stores refund request1902 o at the end of queue 1906, in slot 19080. In response to receivingindication 1909 a from user 120 a, indicating that user 120 a isavailable to process a refund request, virtual store tool 405 assignsthe refund request 1902 a that is stored in the first slot 1908 a ofqueue 1906 to user 120 a. For example, virtual store tool 405 generatesa graphical user interface on device 115 a of user 120 a, for use byuser 120 a in processing refund request 1902 a, as described in furtherdetail below, in the discussion of FIG. 19C. Virtual store tool 405 thenmoves each refund request 1902 b through 1902 o that remains in queue1906 up one slot, such that refund request 1902 b will be the nextrefund request assigned to a user 120, once a user 120 indicates thathe/she is available to process a refund request. Each refund request1902 a through 1902 o submitted by customers 105 may include anysuitable information. For example, as illustrated in FIG. 19A, incertain embodiments, in addition to an identification of the physicalitem(s) 315 w for which customer 105 a is submitting refund request 1902a, the request may also include a description 1904 of the reasoncustomer 105 a submitted the refund request. For instance, description1904 may indicate: (1) that the customer did not select a specific item315 w; (2) that the customer did not select a specific item 315 w butdid select a different item for which he/she was not charged; (3) thatthe customer was charged for a greater quantity of an item 315 w thanwhat he/she selected; and/or (4) any other suitable information that mayaid virtual store tool 405 in processing the customer's refund request.In certain embodiments, description 1904 is a free-form textualdescription submitted by customer 105 a. For example, in someembodiments, customer 105 a may enter description 1904 in a text boxdisplayed on his/her mobile device 125 a. In certain embodiments,description 1904 is one of a number of standard statements that customer105 a may choose from amongst. For example, in some embodiments,customer 105 a may enter description 1904 by selecting from a number ofoptions displayed as a drop-down list on his/her mobile device 125 a.

In some embodiments, refund request 1902 a submitted by customer 105 aincludes information identifying the shopping session associated withthe refund request and/or a portion of the shopping session relevant tothe refund request. As an example, in certain embodiments, the refundrequest includes the start and/or end times of the shopping session. Forexample, refund request 1902 a may indicate that the shopping sessionbegan at 4:15:05 pm on Jun. 26, 2020 and lasted until 4:17:55 pm thatsame day. As another example, in certain embodiments, the refund requestincludes an identification number associated with the shopping session,which virtual store tool 405 may use to determine the start and/or endtimes of the shopping session. For example, virtual store tool 405 mayuse the identification number to look up the start and/or end times ofthe shopping session in the set of shopping session identificationinformation 486 stored in database 482. As a further example, in certainembodiments, refund request includes an identification number associatedwith the shopping session, which virtual store tool 405 may use, inconjunction with the items included in the refund request, to determineportions of the shopping session associated with the refund request. Forexample, virtual store tool 405 may use the identification number, alongwith the items included in the refund request, to look up the times ofthe shopping session associated with the items (e.g., the times at whichthe items were added to a virtual shopping cart during a virtualemulation of the shopping session and/or the times at which algorithm488 determined that the items were selected during the shopping session)in the set of shopping session identification information 486 stored indatabase 482.

b. Processing a Refund Request

In response to assigning a user 120 a a refund request 1902 a, virtualstore tool 405 may generate a graphical user interface for display ondisplay 410 of device 115 a, which user 120 a may use to process therefund request. FIG. 19C presents an example of a graphical userinterface 1900 that may be used by user 120 a to process refund request1902 a. In certain embodiments, graphical user interface 1900 may beused by a user 120 to review a prior shopping session of customer 105 inphysical store 100 and/or a portion of a prior shopping session ofcustomer 105, in order to process the refund request submitted by thecustomer.

i. Displaying Information Received in the Refund Request

As illustrated in FIGS. 19A and 19C, in response to receiving a refundrequest 1902 a, virtual store tool 405 may display informationassociated with the request on graphical user interface 1900. Thisinformation may aid user 120 a in processing the refund request. Forexample, virtual store tool 405 may display description 1904 receivedfrom customer 105 a in region 1910 of graphical user interface 1900and/or an identification of the item(s) 320 w for which customer 105 ais requesting the refund in region 1914 of graphical user interface1900. As described above description 1904 may include text submitted bycustomer 105 a detailing the reason that he/she submitted the refundrequest.

ii. Video-Based Review of the Shopping Session Associated with theRefund Request

A. Identifying and Displaying Video of the Shopping Session

In response to receiving refund request 1902 a, virtual store tool 405additionally identifies and displays historical video segments 1922 athrough 1922 f on graphical user interface 1900. Historical videosegments 1922 a through 1922 f correspond to segments of video capturedby layout cameras 490 and/or rack cameras 495 during the shoppingsession of customer 105 a that is associated with refund request 1902 a.In certain embodiments, historical video segments 1922 a through 1922 fcorrespond to segments of video captured by layout cameras 490 and/orrack cameras 495 during the full duration of the shopping session ofcustomer 105 a. In some embodiments, historical video segments 1922 athrough 1922 f correspond to segments of video captured by layoutcameras 490 and/or rack cameras 495 during a portion of the shoppingsession of customer 105 a.

Virtual store tool 405 may obtain video segments 1922 a through 1922 fin any suitable manner. As an example, in certain embodiments virtualstore tool 405 may use information provided in refund request 1902 a toobtain video segments 1922 a through 1922 f. As an example, in certainembodiments refund request 1902 a includes the start and/or end times ofthe shopping session. In some embodiments, the refund request includesan identification number for the shopping session, which may be used tolook up the start and/or end times of the shopping session in the set ofshopping session identification information 486 stored in database 482.Once virtual store tool 405 has obtained the start and/or end times ofthe shopping session, the tool may use these times to extract videosegments 1922 a through 1922 f from videos 484 stored in database 482.For example, virtual store tool 405 may store a portion of video 484 abeginning at a timestamp corresponding to the start time and ending at atimestamp corresponding to the end time as video segment 1922 a. Virtualstore tool 405 may generate each of video segments 1922 b through 1922 fin a similar manner. As another example, in certain embodiments, refundrequest 1902 a includes an identification number for the shoppingsession, which virtual store tool 405 may use, in conjunction with theitem(s) 315 w for which refund request 1902 a was submitted, to identifyportion(s) of the shopping session relevant to evaluating whether or notcustomer 105 selected item(s) 315 w. For example, virtual store tool 405may use the identification number, along with the item(s) 315 w includedin the refund request, to look up the time(s) of the shopping sessionassociated with the item(s) (e.g., the time(s) at which the item(s) wereadded to a virtual shopping cart during a virtual emulation of theshopping session and/or the time(s) at which algorithm 488 determinedthat the item(s) were selected during the shopping session) in the setof shopping session identification information 486 stored in database482. Once virtual store tool 405 has obtained the time(s) of theshopping session, the tool may use these time(s) to extract videosegments 1922 a through 1922 f from videos 484 stored in database 482.For example, virtual store tool 405 may store, as video segment 1922 a,a portion of video 484 a beginning at a timestamp occurring a set timeinterval before the time of the shopping session that is associated withcustomer 105 a selecting item 315 w (as determined by algorithm 488and/or a virtual emulation of the customer's shopping session usingvirtual store tool 405) and ending at a timestamp occurring a set timeinterval after the time of the shopping session that associated with thecustomer selecting item 315 w as video segment 1922 a. Virtual storetool 405 may generate each of video segments 1922 b through 1922 f in asimilar manner.

B. Using Video of the Shopping Session to Process the Refund Request

User 120 a may use graphical user interface 1900, generated by virtualstore tool 405, to process a refund request 1902 a submitted by acustomer 105 a by reviewing video segments 1922 a through 1922 f todetermine whether any of these video segments include video of customer105 a selecting one or more physical items for which the customersubmitted the refund request. For example, with respect to the exampleillustrated in FIGS. 19A and 19C, user 120 a may review video segments1922 a through 1922 f to determine whether any of these video segmentsdepict customer 105 a selecting physical item 315 w (which correspondsto virtual item 320 w).

Virtual store tool 405 may provide graphical user interface 1900 withany of a number of features that may aid user 120 a in processing refundrequest 1902 a. As a first example, in certain embodiments, and in asimilar manner as that described in Section III above for camera feedsegments 415, virtual store tool 405 may assign a common slider bar 705to each of video segments 1922 a through 1922 f. Slider 710 on sliderbar 705 is configured to control the playback process of the videosegment. In certain embodiments, the playback of each video segment 1922a through 1922 f is synchronized with that of the other video segments1922 a through 1922 f, such that an adjustment of the slider 710 on anyof the copies of slider bar 705 leads to a corresponding adjustment ofthe playback progress of all of the displayed video segments 1922 athrough 1922 f.

In certain embodiments, in order to further aid user 120 a in reviewingthe shopping session associated with refund request 1902 a, slider bar705 may include one or more markers 1916. For example, as illustrated inFIG. 19C, slider bar 705 may include marker 1916, located at a markerposition on slider bar 705 and corresponding to a marker playback time.Each marker 1916 may be associated with an event related to an item 315w/320 w that is a subject of the refund request. For example, asillustrated in FIG. 19C, marker 1916 may be associated with an eventrelated to item 320 w. As an example of an event that marker 1916 may beassociated with, in certain embodiments in which virtual store tool 405was previously used to emulate the shopping session of customer 105 aand generate the receipt 1465 a associated with the customer's refundrequest, the marker playback time associated with marker 1916 maycorrespond to the playback time of camera segments captured by layoutcameras 490 and/or rack cameras 495, during the previous virtualemulation session, at which point a user added virtual item 320 w tovirtual shopping cart 420. As another example, the marker playback timeassociated with marker 1916 may indicate the historical time at whichalgorithm 488 added physical item 315 w to algorithmic shopping cart1420 during customer 105's previous shopping session. Accordingly,marker 1916 may indicate to user 120 a the general time of videosegments 1922 a through 1922 f around which time user 120 a might expectto view customer 105 a selecting item 315 w, if customer 105 did in factselect item 315 w.

As a second example of a feature that may aid user 120 a in processingrefund request 1902 a, in order to aid user 120 a in observing customer105 a on video segments 1922 a through 1922 f, in certain embodiments,user 120 a can choose to display any of video segments 1922 a through1922 f as enlarged video segment 1924. For example, as illustrated inFIG. 19C, user 120 a may choose to display historical video segment 1922c as enlarged video segment 1924. User 120 a can select one of videosegments 1922 a through 1922 f to display as enlarged video segment 1924in any suitable manner. For example, in certain embodiments, user 120 acan click on a given video segment 1922 a through 1922 f to instructvirtual store tool 405 to display the video segment as enlarged videosegment 1924.

As a third example of a feature that may aid user 120 a in processingrefund request 1902 a, in certain embodiments, in order to aid user 120a in observing customer 105 a's interactions with a given physical rack215 a, virtual store tool 405 may display a video segment captured by arack camera 495, directed at the given physical rack 215 a, as enlargedvideo segment 1924. As an example, in certain embodiments, virtual storetool 405 may display a video segment captured by a rack camera 495directed at the physical rack 210 a that stores a physical item 315 wthat is a subject of customer 105 a's refund request 1902 a. Forinstance, in certain embodiments, virtual store tool 405 mayautomatically display, as enlarged video segment 1924, a video segmentcaptured by a rack camera 495 directed at the physical rack 210 a thatstores physical item 315 w. In some embodiments, virtual store tool 405may display, as enlarged video segment 1924, a video segment captured bya rack camera 495 directed at the physical rack 210 a that storesphysical item 315 w, in response to user 120 a selecting (e.g., clickingon, double-clicking on, and/or any other suitable method of selecting)the corresponding virtual item 320 w from region 1914 of graphical userinterface 1900.

As a fourth example of a feature that may aid user 120 a in processingrefund request 1902 a, in certain embodiments, virtual store tool 405may highlight a virtual rack 230 a, corresponding to a physical rack 210a that stores a first physical item 315 w that is a subject of customer105 a's refund request. Highlighting virtual rack 230 a may aid user 120a in identifying which video segments of the shopping session he/sheshould review. For example, in response to observing that virtual rack230 a is highlighted, user 120 a may choose to display a video segmentcaptured by a rack camera 495 that is directed at the correspondingphysical rack 210 a, by selecting from amongst the available rackcameras 495 that were directed at physical rack 210 a. For instance, incertain embodiments, virtual store tool 405 may display icons 1706 a/bon virtual layout 205, where the locations of icons 1706 a/b on virtuallayout 205 emulate the physical locations of rack cameras 495 withinphysical store 100. In response to user 120 a selecting an icon 1706 a/b(e.g., clicking on the icon, double-clicking on the icon, and/or anyother suitable method of selecting the icon), virtual store tool 405 maydisplay, as enlarged video segment 1924, a video segment captured by therack camera 495 corresponding to the icon 1706 a/b. Virtual store tool405 may highlight a virtual rack 230 a that stores a virtual item 320 wthat is a subject of customer 105 a's refund request 1902 a in anysuitable manner. As an example, in certain embodiments, virtual storetool 405 may automatically highlight virtual rack 230 a in response todisplaying graphical user interface 1900 on display 410. As anotherexample, virtual store tool 405 may highlight virtual rack 230 a inresponse to user 120 a selecting (e.g., clicking on, double-clicking on,and/or any other suitable method of selecting) a virtual item 320 w fromregion 1920 of graphical user interface 1900, which is stored on virtualrack 230 a.

User 120 a may accept or reject a customer's refund request 1902 a basedon the user's review of video segments 1922 a through 1922 f and/or1924. As an example, in certain embodiments, user 120 a may observe, invideo segments 1922 a through 1922 f and/or 1924, customer 105 aselecting a physical item 315 w that is a subject of customer 105 a'srefund request 1902 a. Accordingly, user 120 a may decline customer 105a's refund request for that physical item 315 w. User 120 a may declinecustomer 105 a's refund request 1902 a in any suitable manner. Forexample, in certain embodiments, user 120 a may select decline button1918 to decline the refund request. In response to user 120 a decliningthe refund request, virtual store tool 405 may transmit a message 1926(illustrated in FIG. 19A) to customer 105 a indicating that his/herrefund request 1902 a has been denied. For example, in certainembodiments, virtual store tool 405 may transmit a message 1926 todevice 125 a indicating that refund request 1902 a has been denied. Incertain embodiments in which refund request 1902 a includes a requestfor a refund of the purchase price of more than one item, user 120 a maydecline all of a portion of the refund request. For example, in responseto reviewing customer 105 a selecting physical item 315 w in videosegments 1922 a through 1922 f and/or 1924, user 120 a may decline afirst portion of the refund request associated with a first item 315 w,but nevertheless accept a second portion of the refund requestassociated with a second item.

As another example, in some embodiments, user 120 a may determine, byobserving video segments 1922 a through 1922 f and/or 1924, thatcustomer 105 a did not select a physical item 315 w that is a subject ofcustomer 105 a's refund request 1902 a. Accordingly, user 120 a mayaccept customer 105 a's refund request for that physical item 315 w.User 120 a may accept customer 105 a's refund request in any suitablemanner. For example, in certain embodiments, user 120 a may selectaccept button 1920 to accept the refund request. In response to user 120a accepting customer 105 a's refund request, virtual store tool 405 maycredit an account belonging to customer 105 a with the purchase price ofthe physical item 315 w. Virtual store tool 405 may also transmit amessage 1926 (illustrated in FIG. 19A) to customer 105 a indicating thathis/her refund request has been accepted. For example, in certainembodiments, virtual store tool 405 may transmit a message 1926 todevice 125 a indicating that refund request 1902 a has been accepted.

iii. Image-Based Review of the Shopping Session Associated with theRefund Request

In certain embodiments, in addition to or instead of reviewing videosegments 1922 a through 1922 f and/or 1924, user 120 a may review animage 1710 of customer 105 a that was captured as the customer exitedphysical store 100, in order to process a refund request 1902 asubmitted by customer 105 s. Image 1710 may have been captured in anysuitable manner. For example, in certain embodiments in which physicalstore 100 includes turnstiles 510 to control the entry/exit of persons105 into/out of the store (as depicted in FIG. 5A), physical store 100may include a camera configured to capture an image 1710 of customer 105a as customer 105 a exits through a turnstile 510. In certainembodiments, image 1710 is stored in shopping session identificationinformation 486 in database 482. In response to customer 105 asubmitting a refund request 1902 a that includes information identifyingthe shopping session, virtual store tool 405 may access database 482 anduse this identification information to locate image 1710. Virtual storetool 405 may then display image 1710 on graphical user interface 1900.User 120 a may review image 1710 to quickly determine whether customer105 a exited physical store 100 with any of the physical items that arethe subjects of the customer's refund request. For example, asillustrated in FIG. 19, user 120 a may review image 1710 to determinethat customer 105 a exited physical store 100 with physical item 315 jbut not physical item 315 w. Accordingly, user 120 a may accept customer105 a's refund request for a refund of the purchase price of physicalitem 315 w.

c. Method for Processing a Refund Request

FIG. 20 presents a flowchart (described in conjunction with elements ofFIGS. 19A through 19C) illustrating the manner by which virtual storetool 405 may be used to process a refund request 1902 a submitted by acustomer 105 a. In step 2002 virtual store tool 405 receives a request1902 a for a refund of the purchase price(s) of one or more physicalitems 315 w that were charged to an account belonging to customer 105 ain response to a prior shopping session conducted by customer 105 a inphysical store 100. In step 2004 virtual store tool 405 identifies theshopping session associated with the refund request 1902 a. For example,in certain embodiments, the refund request submitted by customer 105 aincludes information identifying the shopping session associated withthe refund request. Accordingly, virtual store tool 405 may access thisinformation to identify the shopping session associated with the refundrequest. As an example, in certain embodiments, refund request 1902 aincludes the start and/or end times of the shopping session. Forexample, the refund request may indicate that the shopping session beganat 4:15:05 pm on Jun. 26, 2020 and lasted until 4:17:55 pm that sameday. As another example, in certain embodiments, refund request 1902 aincludes an identification number associated with the shopping session.Virtual store tool 405 may use this identification number to determinethe start and/or end times of the shopping session. For example, virtualstore tool 405 may use the identification number to look up the startand/or end times of the shopping session in the set of shopping sessionidentification information 486 stored in database 482. As a furtherexample, in certain embodiments, refund request 1902 a includes anidentification number associated with the shopping session, whichvirtual store tool 405 may use, in conjunction with the items includedin the refund request, to determine portions of the shopping sessionassociated with the refund request. For example, virtual store tool 405may use the identification number, along with the items included in therefund request, to look up the times of the shopping session associatedwith the items (e.g., the times at which the items were added to avirtual shopping cart during a virtual emulation of the shopping sessionand/or the times at which algorithm 488 determined that the items wereselected during the shopping session) in the set of shopping sessionidentification information 486 stored in database 482.

In step 2006 virtual store tool 405 generates and displays historicalvideo segments 1922 a through 1922 f and/or 1924 on graphical userinterface 1900. Historical video segments 1922 a through 1922 f and/or1924 correspond to segments of video captured by layout cameras 490and/or rack cameras 495 during the shopping session of customer 105 athat is associated with the refund request. Virtual store tool 405 mayobtain video segments 1922 a through 1922 f and/or 1924 in any suitablemanner. As an example, in certain embodiments virtual store tool 405uses the start and/or end times associated with the shopping session,which were obtained from the refund request, to extract video segments1922 a through 1922 f and/or 1924 from videos 484 a through 484 h storedin database 482. For example, virtual store tool 405 may store a portionof video 484 a beginning at a timestamp corresponding to the start timeand ending at a timestamp corresponding to the end time as video segment1922 a. Virtual store tool 405 may generate each of video segments 1922b through 1922 f and/or 1924 in a similar manner. As another example, incertain embodiments, virtual store tool 405 uses the time(s) of theshopping session associated with the item(s) included in refund request1902 a to extract video segments 1922 a through 1922 f and/or 1924 fromvideos 484 stored in database 482. For example, virtual store tool 405may store, as video segment 1922 a, a portion of video 484 a beginningat a timestamp occurring a set time interval before the time of theshopping session that is associated with customer 105 a selecting item315 w (as determined by algorithm 488 and/or a virtual emulation of thecustomer's shopping session using virtual store tool 405) and ending ata timestamp occurring a set time interval after the time of the shoppingsession that associated with the customer selecting item 315 w as videosegment 1922 a. Virtual store tool 405 may generate each of videosegments 1922 b through 1922 f and/or 1924 in a similar manner.

In step 2008 virtual store tool 405 selects a first physical item 315 w(corresponding to virtual item 320 w) from the refund request. In step2010 virtual store tool 405 determines whether customer 105 a selectedphysical item 315 w during the shopping session associated with therefund request. For example, in certain embodiments, virtual store tool405 receives an indication from user 120 a that user 120 a eitherobserved customer 105 a selecting physical item 315 w on video segments1922 a through 1922 f and/or 1924 or did not observe customer 105 aselecting physical item 315 w on video segments 1922 a through 1922 fand/or 1924. If, in step 2010 virtual store tool 405 determines thatcustomer 105 a selected physical item 315 w during the shopping session,in step 2012 virtual store tool 405 declines to process the refundrequest associated with physical item 315 w. In certain embodiments,virtual store tool 405 may transmit a message 1926 to customer 105 ainforming customer 105 a that his/her refund request has been denied.Method 2000 then proceeds to step 2016. If, in step 2010 virtual storetool 405 determines that customer 105 a did not select physical item 315w during the shopping session, in step 2014 virtual store tool 405processes the refund request associated with physical item 315 w. Forexample, in certain embodiments, virtual store tool 405 may credit anaccount belonging to customer 105 a with the purchase price of thephysical item 315 w. In some embodiments, virtual store tool 405 mayalso transmit a message 1926 to customer 105 a indicating that his/herrefund request 1902 a has been accepted. Method 2000 then proceeds tostep 2016.

In step 2016 virtual store tool 405 determines whether customer 105 a'srefund request 1902 a includes any additional items 315 w. If, in step2016 virtual store tool 405 determines that customer 105 a's refundrequest includes one or more additional items, method 2000 returns tostep 2008.

Modifications, additions, or omissions may be made to method 2000depicted in FIG. 20. Method 2000 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as virtual store tool 405 (or componentsthereof) performing the steps, any suitable component of system 400,such as device(s) 115 for example, may perform one or more steps of themethod.

IX. Selective Use of Virtual Emulation as a Means for VerifyingAlgorithmic Shopping Carts

As described in Section VI above, in certain embodiments, virtual storetool 405 may be used in conjunction with an external algorithm 488 thatis configured to model a customer's shopping session in physical store100, based on inputs received from sensors 498 located within the store.For example, virtual store tool 405 may be used to verify thedeterminations made by algorithm 488 of the items that customer 105selected during the shopping session and/or to help improve the accuracyof algorithm 488 by providing feedback to algorithm 488. In certainembodiments, using virtual store tool 405 to verify each determinationmade by algorithm 488 may not be feasible. For example, significantprocessing resources may be expended by verifying each determinationmade by algorithm 488. Accordingly, rather than verifying eachdetermination made by algorithm 488, it may be desirable to selectivelyverify determinations made by algorithm 488. For example, it may bedesirable to verify those determinations made by algorithm 488 for whichthere exists a reasonable probability that the algorithmic determinationis incorrect, while declining to verify those determinations made byalgorithm 488 for which there exists a high probability that thealgorithmic determination is correct.

a. Selective Verification System Overview

FIG. 21 presents an example system 2100 configured to use virtual storetool 405 to verify those determinations made by algorithm 488, for whichthe probability that the algorithmic determination is incorrect isgreater than a threshold. In particular, cart review assistant 2102 ofsystem 2100 is configured to determine whether or not to invoke virtualstore tool 405 to verify all or a portion of an algorithm shopping cart1420 (which includes items that algorithm 488 determined that customer105 selected during a shopping session in physical store 100), based onthe probability that one or more of the items included in thealgorithmic shopping cart 1420 is incorrect. As can be seen by acomparison between system 2100 presented in FIG. 21 and system 400presented in FIG. 4, system 2100 includes many of the same components assystem 400—namely virtual store tool 405, network 430, external system485, layout cameras 490, rack cameras 495, and sensors 498. Accordingly,in the discussion that follows, it is assumed that the features andfunctions of these shared components include any of thosefeatures/functions presented in Sections I through VII above.

As illustrated in FIG. 21, in addition to virtual store tool 405,network 430, external system 485, layout cameras 490, rack cameras 495,and sensors 498, system 2100 includes cart review assistant 2102 and, incertain embodiments, database 2104. Cart review assistant 2102 includesa processor 2108 and a memory 2110. This disclosure contemplatesprocessor 2108 and memory 2110 being configured to perform any of thefunctions of cart review assistant 2102 described herein. For example,memory 2110 may include a set of instructions 2124 that, when executedby processor 2108, perform one or more of the functions of cart reviewassistant 2102 described herein. Generally, cart review assistant 2102is configured to apply machine learning algorithm 2112 to sensor inputs2122, to determine whether or not all or a portion of algorithmicshopping cart 1420, generated by external system 485, is likely toinclude errors (i.e., likely to not accurately reflect the itemsactually selected by customer 105 during his/her shopping session) andis therefore a good candidate for verification by virtual store tool405. For example, machine learning algorithm 2112 may determine whetherthe probability that algorithmic shopping cart 1420 includes an error isgreater than a threshold. In certain embodiments, cart review assistant2102 is also configured use training data 2106, stored in database 2104,to train machine learning algorithm 2112 to identify virtual shoppingcarts 1420 as candidates for verification. In some embodiments, cartreview assistant 2102 is further configured to update/refine machinelearning algorithm 2112 based on feedback 2116 received from virtualstore tool 405 for prior decisions made by the algorithm, in order toimprove the accuracy of machine learning algorithm 2112. These functionsof cart review assistant 2102 are described in further detail below, andin particular, in the discussion of FIGS. 22 and 23.

Processor 2108 is any electronic circuitry, including, but not limitedto central processing units (CPUs), graphics processing units (GPUs),microprocessors, application specific integrated circuits (ASIC),application specific instruction set processor (ASIP), and/or statemachines, that communicatively couples to memory 2110 and controls theoperation of cart review assistant 2102. Processor 2108 may be 8-bit,16-bit, 32-bit, 64-bit or of any other suitable architecture. Processor2108 may include an arithmetic logic unit (ALU) for performingarithmetic and logic operations, processor registers that supplyoperands to the ALU and store the results of ALU operations, and acontrol unit that fetches instructions from memory and executes them bydirecting the coordinated operations of the ALU, registers and othercomponents. Processor 2108 may include other hardware and software thatoperates to control and process information. Processor 2108 executessoftware stored on memory to perform any of the functions describedherein. Processor 2108 controls the operation and administration of cartreview assistant 2102 by processing information received from virtualstore tool 405, network 430, external system 485, layout cameras 490,rack cameras 495, sensors 498, and/or database 2104. Processor 2108 maybe a programmable logic device, a microcontroller, a microprocessor, anysuitable processing device, or any suitable combination of thepreceding. Processor 2108 is not limited to a single processing deviceand may encompass multiple processing devices.

Memory 2110 may store, either permanently or temporarily, data,operational software, or other information for processor 2108. Memory2110 may include any one or a combination of volatile or non-volatilelocal or remote devices suitable for storing information. For example,memory 2110 may include random access memory (RAM), read only memory(ROM), magnetic storage devices, optical storage devices, or any othersuitable information storage device or a combination of these devices.The software represents any suitable set of instructions, logic, or codeembodied in a computer-readable storage medium. For example, thesoftware may be embodied in memory 2110, a disk, a CD, or a flash drive.In particular embodiments, the software may include an applicationexecutable by processor 2108 to perform one or more of the functionsdescribed herein.

Additionally, in certain embodiments, memory 2110 may store machinelearning algorithm 2112. Machine learning algorithm 2112 is anyalgorithm configured to determine whether algorithmic shopping cart 1420is a good candidate for verification by virtual store tool 405 (e.g.,whether algorithmic shopping cart 1420 likely includes errors). Machinelearning algorithm 2112 may determine whether or not algorithmicshopping cart 1420 is a good candidate for verification based on sensorinputs 2122 that were received from sensors 498 located in physicalstore 100, and which were used by algorithm 488 to generate algorithmicshopping cart 1420. In certain embodiments, machine learning algorithm2112 is configured to categorize an algorithmic shopping cart 1420 aseither a good candidate for verification or not a good candidate forverification. Machine learning algorithm 2112 may base itscategorization of an algorithmic shopping cart 1420 as either a goodcandidate or a bad candidate on the probability that the algorithmicshopping cart includes one or more errors. For instance, machinelearning algorithm 2112 may categorize an algorithmic shopping cart 1420as a good candidate for verification by virtual store tool 405 where theprobability that the algorithmic shopping cart includes at least oneerror is greater than a certain threshold.

In some embodiments, machine learning algorithm 2112 may furthercategorize those algorithmic shopping carts 1420 that are goodcandidates for verification according to the probable type and/or causeof the error(s) likely present in the algorithmic shopping cart 1420. Asan example, machine learning algorithm 2112 may categorize analgorithmic shopping cart 1420 that is a good candidate for verificationas likely: (1) including one or more physical items 315 that customer105 did not select; (2) not including one or more physical items 315that customer 105 did select; (3) including an incorrect quantity of oneor more physical items 315 that customer 105 selected; and/or (4)including any other type of error that may be present in algorithmicshopping cart 1420. As another example, machine learning algorithm 2112may categorize an algorithmic shopping cart 1420 that is a goodcandidate for verification as likely including one or more errorsbecause: (1) one or more servers within external system 485 likely wasnot functioning properly; (2) external system 485 likely was notproperly recording information received from one or more weight sensors1300 (illustrated in FIGS. 13B through 13D); (3) one or more weightsensors 1300 likely generated a false positive; (4) algorithm 488 likelygenerated one or more incorrect item counts; (5) algorithm 488 likelyassigned one or more items 315 to the wrong customer 105; (6) algorithm488 likely lost track of customer 105; and/or (7) any other likely causeof the error(s) within algorithmic shopping cart 1420.

Machine learning algorithm 2112 may be configured to classify all or aportion of an algorithmic shopping cart 1420 as a good candidate forverification (e.g., as likely including one or more errors). As anexample, consider an algorithmic shopping cart 1420 that includes twoitems—first item 315 a and second item 315 b. Algorithm 488 may haveadded first item 315 a to algorithmic shopping cart 1420 based on afirst subset of sensor inputs 2122 received from sensors 498 during afirst portion of a shopping session and added second item 315 b toalgorithmic shopping cart 1420 based on a second subset of sensor inputs2122 received from sensors 498 during a second portion of the shoppingsession. In certain embodiments, machine learning algorithm 2112 may beconfigured to classify first item 315 a, based on the first subset ofsensor inputs 2122, as a good candidate for verification by virtualstore tool 405 (e.g., likely incorrect) and to classify second item 315b, based on the second subset of sensor inputs 2122, as not a goodcandidate for verification by virtual store tool 405 (e.g., likelycorrect). In some embodiments, machine learning algorithm 2112 may beconfigured to classify the full algorithmic shopping cart 1420 (e.g.,both first item 315 a and second item 315 b) as a good candidate forverification by virtual store tool 405 (e.g., likely including one ormore errors), based on the full set of sensor inputs 2122. Splitting analgorithmic shopping cart 1420 into portions and determining whether ornot each portion is a good candidate for verification by virtual storetool 405 may be desirable to reduce the computational costs associatedwith using virtual store tool 405 to verify algorithmic shopping cart1420; it may be more efficient to use virtual store tool 405 to review aportion of a shopping session in order to verify a portion of analgorithmic shopping cart 1420 rather than the reviewing the entireshopping session.

Machine learning algorithm 2112 may be any suitable machine learningalgorithm. As a first example, in certain embodiments, machine learningalgorithm 2112 is a supervised learning algorithm. For example, machinelearning algorithm 2112 may be a supervised learning algorithm that hasbeen trained, using training data 2106, to determine whether or not analgorithmic shopping cart 1420 is a good candidate for verificationbased on sensor inputs 2122. Sensor inputs 2122 correspond to inputsthat were received from sensors 498 located in physical store 100 andwere used by algorithm 488 to generate algorithmic shopping cart 1420.As described in detail below, in certain embodiments, training data 2106may include sets of historical sensor inputs 2118 a through 2118 n, eachof which is assigned a corresponding label 2120 a through 2120 nlabeling whether or not an algorithmic shopping cart generated based onthe historical sensor inputs was a good candidate for verification(e.g., whether or not the algorithmic shopping cart likely included oneor more errors). For example, label 2120 a may indicate that thealgorithmic shopping cart generated using historical sensor inputs 2118a was inaccurate, and therefore was a good candidate for verification byvirtual store tool 405. Similarly, label 2120 b may indicate that thealgorithmic shopping cart generated using historical sensor inputs 2118b was accurate and therefore was not a good candidate for verificationby virtual store tool 405. Machine learning algorithm 2112 may betrained, using training data 2106, to identify patterns within the setsof historical sensor inputs 2118 a through 2118 n and correspondinglabels 2120 a through 2120 n, such that machine learning algorithm 2112may assign a label to new sensor inputs 2122, based on these learnedpatterns.

As a specific example of a supervised machine learning algorithm 2112that may be used by cart review assistant 2102 to identify thosealgorithmic shopping carts 1420 that are good candidates forverification by virtual store tool 405, in certain embodiments, machinelearning algorithm 2112 is a neural network algorithm. For example,machine learning algorithm 2112 may be a neural network that includes aninput layer of nodes, one or more hidden layers of nodes, and an outputlayer of nodes. The input layer of nodes is configured to take, asinput, sensor inputs 2122 that are associated with a given shoppingsession of a customer 105 within physical store 100, and to transferthese inputs to the first hidden layer of nodes. Each node within thehidden layer(s) of the neural network receives a certain number ofinputs from the previous layer of nodes and is associated with a set ofweights, one for each received input. Each hidden layer node isconfigured to calculate a weighted sum of the inputs it receives, add abias, and execute an activation function. The output of each hiddenlayer node is passed as input to the next layer of nodes. The outputlayer of nodes receives input from the last hidden layer and generatesthe result of the neural network. For example, in certain embodiments inwhich machine learning algorithm 2112 is configured to classify sensorinputs 2122 as either associated with an algorithmic cart 1420 that is agood candidate for verification (e.g., likely includes one or moreerrors) or an algorithmic cart 1420 that is not a good candidate forverification (e.g., likely does not include one or more errors), theoutput layer may include a single output node, where an output value ofthe node between 0.5 and 1.0 indicates that sensor inputs 2122 areassociated with an algorithmic cart 1420 that is a good candidate forverification and an output value of the node less than 0.5 indicatesthat sensor inputs 2122 are associated with an algorithmic cart 1420that is not a good candidate for verification. Alternatively, the outputlayer may include two nodes—a first node that is used to categorizesensor inputs 2122 as likely associated with an algorithmic cart 1420that is a good candidate for verification, and a second node that isused to categorize sensor inputs 2122 as likely associated with analgorithmic cart 1420 that is not a good candidate for verification.Sensor inputs 2122 are assigned to one of these two categories accordingto which of the two output nodes has the largest value.

In embodiments in which machine learning algorithm 2112 is a neuralnetwork, the neural network may be trained to identify those algorithmicshopping carts 1420 that are good candidates for verification (e.g.,those algorithmic shopping carts 1420 that likely include one or moreerrors) in any suitable manner. For example, in certain embodiments,machine learning algorithm 2112 is trained using training data 2106 byapplying the algorithm to historical sensor inputs 2118, for which thedesired output of the neural network, indicated by labels 2120, isknown. Training the algorithm involves optimizing the values of theweights of the nodes within the network in order to maximize theaccuracy of the neural network (e.g., maximize the likelihood that, whenapplied to a given set of historical sensor inputs 2118, the output ofthe neural network will correspond to the label 2120 assigned to thegiven set of historical sensor inputs 2118). In certain embodiments,cart review assistant 2102 performs this training process.

As a second example of the type of machine learning algorithm 2112 thatmay be used by cart review assistant 2102 to identify those algorithmicshopping carts 1420 that are good candidates for verification by virtualstore tool 405 (e.g., those algorithmic shopping carts 1420 that likelyinclude one or more errors), in certain embodiments, machine learningalgorithm 2112 is a reinforcement learning algorithm, that continues tolearn over time. As a specific example, machine learning algorithm 2112may correspond to a reinforcement learning agent. For each decision madeby reinforcement learning agent 2112, feedback 2116 may be provided tothe agent, in the form of a reward/penalty, and the agent may use thisfeedback 2116 to refine itself in an attempt to maximize future rewards.Rewards/penalties may be provided to the agent as follows: (1) if theagent determines that an algorithmic shopping cart 1420 obtained fromset of sensor inputs 2122 is a good candidate for verification (e.g.,the cart is likely inaccurate) and it is subsequently determined thatthe algorithmic shopping cart 1420 does include one or more errors, afirst reward value is provided to the reinforcement learning agent 2112;(2) if the agent determines that an algorithmic shopping cart 1420obtained from set of sensor inputs 2122 is a good candidate forverification (e.g., the cart is likely inaccurate) and it issubsequently determined that the algorithmic shopping cart 1420 does notinclude any errors, a first penalty value (e.g., a first negative rewardvalue) is provided to the reinforcement learning agent 2112; (3) if theagent determines that an algorithmic shopping cart 1420 obtained fromset of sensor inputs 2122 is not a good candidate for verification(e.g., the cart is likely accurate) and it is subsequently determinedthat the algorithmic shopping cart 1420 does not include any errors, asecond reward value is provided to the reinforcement learning agent2112; and (4) if the agent determines that an algorithmic shopping cart1420 obtained from set of sensor inputs 2122 is a not a good candidatefor verification (e.g., the cart is likely accurate) and it issubsequently determined that the algorithmic shopping cart 1420 does notinclude one or errors, a second penalty value (e.g., a second negativereward value) is provided to the reinforcement learning agent 2112. Incertain embodiments, the reward/penalty values are determined using areward function. In certain embodiments, the first and second rewardvalues are the same and the first and second penalty values are thesame. In some embodiments, the absolute value of the second penaltyvalue is larger than the absolute value of the first penalty value(e.g., reinforcement learning agent 2112 is penalized more heavily fordeterminations that result in incorrect charges to customer 105 than fordeterminations that result in an unnecessary expenditure ofcomputational resources). In certain embodiments, the values of one ormore of the first reward, the second reward, the first penalty, and thesecond penalty, may change, depending on the situation underconsideration by reinforcement learning agent 2112. As an example, alarger reward value may be provided to reinforcement learning agent 2112when the agent determines that a first algorithmic shopping cart 1420 isa good candidate for verification (e.g., the cart is likely inaccurate)and it is subsequently determined that the cart includes error(s) thatwould lead to under/overcharging customer 105 by at least twentydollars, as compared to a situation in which the agent determines that asecond algorithmic shopping cart 1420 is a good candidate forverification and it is subsequently determined that the cart includeserror(s) that would lead to under/overcharging customer 105 by only afew cents. The process of rewarding/penalizing reinforcement learningagent 2112 is described in further detail below, in the discussion ofFIG. 22.

Cart review assistant 2102 may include any suitable reinforcementlearning agent 2112. As an example, in certain embodiments, cart reviewassistant 2102 includes a tabular Q-Learning model that includes a setof state-action pairs, with a q-value assigned to each pair. Each statecorresponds to a set of sensor inputs 2122 and each action correspondsto either sending the algorithmic shopping cart 1420 that is associatedwith the sensor inputs 2122 for verification by virtual store tool 405,or not sending the algorithmic shopping cart 1420 for review. For agiven state, the action with the highest q-value is the action that ischosen by reinforcement learning agent 2112. For example, for a givenset of sensor inputs 2122, reinforcement learning agent 2112 makes thedecision of whether or not to send the associated algorithmic shoppingcart 1420 for review by virtual store tool 405 based on which action(sending the cart for review or not sending the cart for review)corresponds to the highest q-value. Feedback 2116 may be provided toagent 2112 in the form of the reward/penalty values described above. Inresponse to receiving this feedback 2116, agent 2112 may refine theQ-learning model by adjusting the q-values assigned to the state-actionpairs in order to maximize the future rewards received (e.g., tomaximize the Bellman equation). In certain embodiments, sensor inputs2122 are discretized, such that a finite and computationally feasiblenumber of states are possible. For example, in embodiments in whichinputs 2122 include the positions of customer 105 within physical store100, those positions may be specified according to grid points on adiscrete grid overlaying the floor of physical store 100.

As another example of a reinforcement learning agent 2112 that may beused by cart review assistant 2102, in certain embodiments,reinforcement learning agent 2112 is a deep Q network (DQN). Forexample, reinforcement learning agent 2112 may be a deep Q network thatincludes an input layer of nodes, one or more hidden layers of nodes,and an output layer of nodes. The input layer of nodes may be configuredto provide, as input to the network, the set of sensor inputs 2122. Theoutput layer of nodes may be configured to provide, as output, q-valuescorresponding to the actions of sending the algorithmic shopping cart1420 associated with the sensor inputs 2122 for review by virtual storetool 405 or not sensing algorithmic shopping cart 1420 for review byvirtual store tool 405. Feedback 2116 may be provided to the agent 2112in the form of the reward/penalty values described above. In response toreceiving this feedback 2116, the agent may refine itself (e.g., byadjusting the values of weights within the network) in order to maximizethe future rewards it receives (e.g., to maximize the Bellman equation).In certain embodiments, reinforcement learning agent 2112 is a variantof a deep Q network. For example, in some embodiments, machine learningagent 2112 corresponds to a double deep Q network (DDQN). The use of aDDQN algorithm may be desirable to reduce the impact of recency bias onthe algorithm's determinations.

In certain embodiments, system 2100 includes database 2104. Database2104 stores training data 2106. Training data 2106 includes any datathat may be used to train machine learning algorithm 2112. For example,in embodiments in which machine learning algorithm 2112 is a supervisedlearning algorithm, training data 2106 may include sets of historicalsensor inputs 2118 a through 2118 n and labels 2120 a through 2120 n.

Each set of historical sensor inputs 2118 includes inputs received fromsensors 498 (including, for example, cameras 1305 and weight sensors1300) located within physical store 100 that were captured during ashopping session of a customer 105 within physical store 100. Sensors498 may include any sensors located within physical store 100. Forexample, in certain embodiments, sensors 498 may include cameras, lightdetection and range sensors, millimeter wave sensors, weight sensors,and/or any other appropriate sensors, operable to track a customer 105in physical store 100 and to detect information associated with customer105 selecting one or more physical items 315 from physical store 100.

Each label 2120 of labels 2120 a through 2120 n is assigned to acorresponding set of historical sensor inputs 2118 a through 2118 n, andlabels whether the algorithmic shopping cart 1420, generated byalgorithm 488 based on the corresponding set of historical sensor inputs2118, included any errors (and, accordingly, whether algorithmicshopping cart 1420 is a good candidate for verification by virtual storetool 405). As an example, label 2120 a, assigned to set of historicalsensor inputs 2118 a, may indicate that an algorithmic shopping cart1420, generated by algorithm 488 based on set of historical sensorinputs 2118 a, included one or more errors. As another example, label2120 b, assigned to set of historical sensor inputs 2118 b, may indicatethat an algorithmic shopping cart 1420, generated by algorithm 488 basedon set of historical sensor inputs 2118 b, did not include any errors.

In certain embodiments, in addition to labeling sets of historicalsensor inputs 2118 according to whether an algorithmic shopping cart1420, generated by algorithm 488 based on the corresponding set ofhistorical sensor inputs 2118, included any errors, labels 2120 mayfurther label sets of historical sensor inputs 2118 according to theprobable type and/or cause of the error(s) likely present in thealgorithmic shopping cart 1420. As an example, label 2120 a may labelset of historical sensor inputs 2118 a according to whether analgorithmic shopping cart 1420, generated by algorithm 488 based on theset of historical sensor inputs 2118 a: (1) included one or morephysical items 315 that customer 105 did not select; (2) did not includeone or more physical items 315 that customer 105 did select; (3)included an incorrect quantity of one or more physical items 315 thatcustomer 105 selected; and/or (4) included any other type of error. Asanother example, label 2120 a may label set of historical sensor inputs2118 a according to whether an algorithmic shopping cart 1420, generatedby algorithm 488 based on the set of historical sensor inputs 2118included one or more errors because: (1) one or more servers withinexternal system 485 likely was not functioning properly; (2) externalsystem 485 likely was not properly recording information received fromone or more weight sensors 1300; (3) one or more weight sensors 1300likely generated a false positive; (4) algorithm 488 likely generatedone or more incorrect item counts; (5) algorithm 488 likely assigned oneor more items 315 to the wrong customer 105; (6) algorithm 488 likelylost track of customer 105 and/or (7) any other likely cause of theerror(s) within algorithmic shopping cart 1420.

Labels 2120 may be generated in any suitable manner. As an example, incertain embodiments, virtual store tool 405 may generate a label 2120 a,corresponding to historical sensor inputs 2118 a that were generatedduring a given shopping session in physical store 100, in response togenerating a virtual shopping cart 420 for that shopping session andcomparing the virtual shopping cart 420 to an algorithmic shopping cart1420, generated by algorithm 488 based on the historical sensor inputs2118 a. For example, for a given shopping session associated with sensorinputs 2118 a, in response to comparing virtual shopping cart 420 withalgorithmic shopping cart 1420 and determining that the two carts do notmatch, virtual store tool 405 may generate label 2120 a, which indicatesthat the algorithmic shopping cart 1420 associated with sensor inputs2118 a is a good candidate for verification. Similarly, for a givenshopping session associated with sensor inputs 2118 b, in response tocomparing virtual shopping cart 420 with algorithmic shopping cart 1420and determining that the two carts match, virtual store tool 405 maygenerate label 2120 b, which indicates that the algorithmic shoppingcart 1420 associated with sensor inputs 2118 b is not a good candidatefor verification. As another example, in certain embodiments, a user 120may generate labels 2120. For example, user 120 may generate label 2120c based on an analysis of historical sensor inputs 2118 c.

Modifications, additions, or omissions may be made to system 2100without departing from the scope of the invention. For example, system2100 may include any number of users 120, virtual store tools 405,networks 430, external systems 485, layout cameras 490, rack cameras495, sensors 498, and databases 2104. The components may be integratedor separated. Moreover, the operations may be performed by more, fewer,or other components. Additionally, the operations may be performed usingany suitable logic comprising software, hardware, and/or other logic.

b. Operation of the Cart Review Assistant

FIG. 22 illustrates an example process by which machine learningalgorithm 2112 of cart review assistant 2102 (1) determines whether ornot virtual store tool 405 should be used to verify an algorithmicshopping cart 1420 (e.g., whether or not algorithmic shopping cart 1420is likely accurate) and (2) receives feedback 2116 for its decision. Asillustrated in FIG. 22, machine learning algorithm 2112 operates onsensor inputs 2122. Sensor inputs 2122 include inputs received fromsensors 498 located within physical store 100 and were used by algorithm488 to generate algorithmic shopping cart 1420. Sensors 498 may includeany sensors located within physical store 100. For example, in certainembodiments, sensors 498 may include cameras, light detection and rangesensors, millimeter wave sensors, weight sensors, and/or any otherappropriate sensors, operable to track a customer 105 in physical store100 and to detect information associated with customer 105 selecting oneor more physical items 315 from physical store 100.

In response to receiving sensor inputs 2122, machine learning algorithm2112 uses these inputs to determine whether or not the algorithmicshopping cart 1420, obtained by algorithm 488 using sensor inputs 2122,is a good candidate for verification by virtual store tool 405. Asdescribed above, determining whether or not algorithmic shopping cart1420 is a good candidate for verification may include determiningwhether or not a probability that algorithmic shopping cart 1420includes one or more errors is greater than a threshold.

i. Decision to Send an Algorithmic Shopping Cart to Virtual Store Toolfor Review.

In response to determining that algorithmic shopping cart 1420 is a goodcandidate for verification, cart review assistant 2102 sends theshopping session associated with the algorithmic shopping cart 1420 tovirtual store tool 405 for review. Virtual store tool 405 is then usedto virtually emulate the shopping session associated with algorithmicshopping cart 1420, as described in Section IV above. In response tovirtually emulating the shopping session, virtual store tool 405compares the virtual shopping cart 420 that was generated during thevirtual emulation to algorithmic shopping cart 1420, as described inSection VI. Virtual store tool 405 then provides feedback 2116 to cartreview assistant 2102, which indicates whether or not virtual shoppingcart 1420 matched algorithmic shopping cart 420. In certain embodiments,virtual store tool 405 additionally charges an account belonging tocustomer 105 for the purchase price of the items 320 in virtual shoppingcart 420 and transmits a receipt 1465 for the purchase to customer 105.

Cart review assistant 2102 may use the feedback 2116 received fromvirtual store tool 405 in any suitable manner. As an example, in certainembodiments in which machine learning algorithm 2112 is a reinforcementlearning agent, in response to receiving feedback 2116 from virtualstore tool 405, cart review assistant 2102 uses this feedback toreward/punish agent 2112. For instance, in response to receivingfeedback 2116 indicating that virtual shopping cart 1420 and algorithmicshopping cart 420 did not match, cart review assistant 2102 providesreinforcement learning agent 2112 with a positive reward value,rewarding reinforcement learning agent 2112 for deciding to send theshopping session associated with algorithmic shopping cart 1420 tovirtual store tool 405 for review, given that virtual store tool 405determined that algorithmic shopping cart 1420 included at least oneerror. In a similar manner, in response to receiving feedback 2116indicating that virtual shopping cart 1420 matched algorithmic shoppingcart 420, cart review assistant 2102 provides reinforcement learningagent 2112 with a penalty value (e.g., a negative reward value),penalizing reinforcement learning agent 2112 for deciding to send theshopping session associated with algorithmic shopping cart 1420 tovirtual store tool 405 for review (and therefore causing the system toexpend additional processing resources), given that virtual store tool405 determined that algorithm shopping cart 1420 did not include anyerrors.

As another example of the use of feedback 2116 received from virtualstore tool 405, in certain embodiments in which machine learningalgorithm 2112 is a supervised learning algorithm, in response toreceiving feedback 2116 from virtual store tool 405, cart reviewassistant 2102 may use this feedback to generate a label 2120 for sensorinputs 2122. Label 2120 labels whether the algorithmic shopping cart1420, generated by algorithm 488 based on sensor inputs 2122, includedany errors (and, accordingly, whether algorithmic shopping cart 1420 wasa good candidate for verification by virtual store tool 405). Cartreview assistant 2102 may then store label 2120 and sensor inputs 2122in database 2104, thereby adding to training data 2106. As an example,in response to receiving feedback 2116 indicating that virtual shoppingcart 420 did not match algorithmic shopping cart 1420, cart reviewassistant 2102 may generate a label 2120 for sensor inputs 2122indicating that the algorithmic shopping cart 1420, generated byalgorithm 488 based on sensor inputs 2122, included at least one error(and, accordingly is a good candidate for verification by virtual storetool 405). As another example, in response to receiving feedback 2116indicating that virtual shopping cart 420 matched algorithmic shoppingcart 1420, cart review assistant 2102 may generate a label 2120 forsensor inputs 2122 indicating that the algorithmic shopping cart 1420,generated by algorithm 488 based on sensor inputs 2122, did not includeany errors (and, accordingly is not a good candidate for verification byvirtual store tool 405). In certain embodiments, cart review assistant2102 may use training data 2106 to retrain machine learning algorithm2112. For example, in some embodiments, cart review assistant 2102 mayuse training data 2106 to retrain machine learning algorithm 2112 atregular intervals.

ii. Decision not to Send an Algorithmic Shopping Cart to Virtual StoreTool for Review.

In response to determining that algorithmic shopping cart 1420 is not agood candidate for verification, in certain embodiments, cart reviewassistant 2102 charges an account belonging to customer 105 for thepurchase price of the items 320 in algorithmic shopping cart 1420 andtransmits a receipt 1465 to customer 105. In some embodiments, cartreview assistant 2102 instructs another component of system 2100, suchas virtual store tool 405, external system 485, or any other suitablecomponent, to perform such tasks.

Feedback 2116 for a decision by machine learning algorithm 2112 not tosend the shopping session associated with algorithmic shopping cart 1420to virtual store tool 405 may be generated in any suitable manner. As anexample, in certain embodiments, the feedback 2116 for a decision bymachine learning algorithm 2112 not to send the shopping sessionassociated with algorithmic shopping cart 1420 to virtual store tool 405for review may originate from customer 105, in the form of customerfeedback 2208. For example, in certain embodiments, in response toreceiving receipt 1465, customer 105 may submit a refund request 2208 tovirtual store tool 405, disputing one or more charges, as described inSection VII above. In response to receiving the refund request, virtualstore tool 405 may determine whether or not to issue a refund to anaccount belonging to customer 105, as described in detail in Section VIIabove. If virtual store tool 405 determines to issue a refund tocustomer 105, this may indicate that the algorithmic shopping cart 1420that was used to charge the account belonging to customer 105, includedone or more errors. Accordingly, virtual store tool 405 may provide thisinformation to cart review assistant 2102 as feedback 2116. On the otherhand, if virtual store tool 405 determines not to issue a refund tocustomer 105, this may indicate that the algorithmic shopping cart 1420,used to charge the account belonging to customer 105, did not includeany errors. Accordingly, virtual store tool 405 may provide thisinformation to cart review assistant 2102 as feedback 2116.

As another example of the manner by which feedback 2116 may begenerated, in certain embodiments, the feedback 2116 for a decision bymachine learning algorithm 2112 not to send the shopping sessionassociated with algorithmic shopping cart 1420 to virtual store tool 405for review may originate directly from virtual store tool 405. Forexample, in certain embodiments, a customer 105 may be unlikely tosubmit a refund request 2208 when an error in algorithmic shopping cart1420 results in system 2100 undercharging customer 105 for his/hershopping session. Accordingly, in order to obtain feedback 2116 relatedto such shopping sessions, in certain embodiments, cart review assistant2102 may randomly decide to send a shopping session associated with analgorithmic shopping cart 1420 to virtual store tool 405 for review,even though machine learning algorithm 2112 determined that the shoppingsession was not a good candidate for review. Virtual store tool 405 maythen be used to virtually emulate the shopping session associated withalgorithmic shopping cart 1420, as described in Section IV above. Inresponse to virtually emulating the shopping session, virtual store tool405 compares the virtual shopping cart 420 that was generated during thevirtual emulation to algorithmic shopping cart 1420, as described inSection VI. Virtual store tool 405 then provides feedback 2116 to cartreview assistant 2102, indicating whether or not virtual shopping cart1420 matched algorithmic shopping cart 420. In certain embodiments,virtual store tool 405 additionally charges an account belonging tocustomer 105 for the purchase price of the items 320 in virtual shoppingcart 420 and transmits a receipt 1465 for the purchase to customer 105.

Cart review assistant 2102 may use the feedback 2116 received fromcustomer 105 and/or virtual store tool 405 in any suitable manner. As anexample, in certain embodiments in which machine learning algorithm 2112is a reinforcement learning agent, in response to receiving feedback2116 from virtual store tool 405, cart review assistant 2102 uses thisfeedback to reward/punish agent 2112. As an example, in response toreceiving feedback 2116 indicating that virtual shopping cart 1420 andalgorithmic shopping cart 420 did not match, cart review assistant 2102provides reinforcement learning agent 2112 with a punishment value(e.g., a negative reward value), punishing reinforcement learning agent2112 for not deciding to send the shopping session associated withalgorithmic shopping cart 1420 to virtual store tool 405 for review (andtherefore relying on algorithmic shopping cart 1420 to generate receipt1465), given that virtual store tool 405 determined that algorithmicshopping cart 1420 included at least one error. As another example, inresponse to receiving feedback 2116 indicating that virtual shoppingcart 1420 matched algorithmic shopping cart 420, cart review assistant2102 provides reinforcement learning agent 2112 with a reward value,rewarding reinforcement learning agent 2112 for deciding not to send theshopping session associated with algorithmic shopping cart 1420 tovirtual store tool 405 for review (and therefore conserving processingresources that would otherwise have been expended), given that virtualstore tool 405 determined that algorithm shopping cart 1420 did notinclude any errors.

As another example of the use of feedback 2116 received from customer105 and/or virtual store tool 405, in certain embodiments in whichmachine learning algorithm 2112 is a supervised learning algorithm, inresponse to receiving feedback 2116 from virtual store tool 405, cartreview assistant 2102 may use this feedback to generate a label 2120 forsensor inputs 2122. As an example, in response to receiving feedback2116 indicating that virtual shopping cart 420 did not match algorithmicshopping cart 1420, cart review assistant 2102 may generate a label 2120for sensor inputs 2122 indicating that the algorithmic shopping cart1420, generated by algorithm 488 based on sensor inputs 2122, includedat least one error (and, accordingly is a good candidate forverification by virtual store tool 405). As another example, in responseto receiving feedback 2116 indicating that virtual shopping cart 420matched algorithmic shopping cart 1420, cart review assistant 2102 maygenerate a label 2120 for sensor inputs 2122 indicating that thealgorithmic shopping cart 1420, generated by algorithm 488 based onsensor inputs 2122, did not include any errors (and, accordingly is nota good candidate for verification by virtual store tool 405). Asdescribed above, in certain embodiments, cart review assistant 2102 mayuse training data 2106 to retrain machine learning algorithm 2112.

c. Method for Selectively Validating Algorithmic Shopping Carts

FIG. 23 presents a flowchart (described in conjunction with elements ofFIGS. 21 and 22) illustrating the manner by which cart review assistant2102 uses machine learning algorithm 2112 to determine whether or notvirtual store tool 405 should be used to verify an algorithmic shoppingcart 1420 (e.g., whether or not algorithmic shopping cart 1420 is likelyaccurate) and receives feedback 2116 for this decision. In step 2302cart review assistant 2102 receives sensor inputs 2122 associated with ashopping session of a customer 105 in physical store 100. In certainembodiments, sensor inputs 2122 were used by algorithm 488 to generatean algorithmic shopping cart 1420 for the shopping session of customer105. In step 2304 cart review assistant 2102 applies machine learningalgorithm 2112 to sensor inputs 2122 to determine whether or notalgorithmic shopping cart 1420 is a good candidate for review.

In step 2306 cart review assistant 2102 determines whether machinelearning algorithm 2112 determined that algorithmic shopping cart 1420is a good candidate for review (e.g., whether algorithmic shopping cart1420 likely includes one or more errors). If, in step 2306 cart reviewassistant 2102 determines that machine learning algorithm 2112determined that algorithmic shopping cart 1420 is a good candidate forreview, in step 2308 cart review assistant 2102 instructs virtual storetool 405 to virtually emulate the shopping session of customer 105 thatis associated with sensor inputs 2122. This virtual emulation mayinclude (1) generating a virtual shopping cart 420 associated with theshopping session, (2) using virtual shopping cart 420 to charge anaccount belonging to customer 105 for the selections he/she made duringthe shopping session, and (3) issuing a receipt 1465 to customer 105. Instep 2310 virtual store tool 405 determines whether the virtual shoppingcart 420 matches the algorithmic shopping cart 1420. If, in step 2310virtual store tool 405 determines that virtual shopping cart 420 doesnot match algorithmic shopping cart 1420, in step 2312 virtual storetool 405 provides feedback 2116 to cart review assistant 2102,indicating that virtual shopping cart 420 does not match algorithmicshopping cart 1420. In certain embodiments in which machine learningalgorithm 2112 is a reinforcement learning agent, cart review assistant2102 then uses this feedback to provide a reward to reinforcementlearning agent 2112. Method 2300 then proceeds to step 2326. If, in step2310 virtual store tool 405 determines that virtual shopping cart 420matches algorithmic shopping cart 1420, in step 2314 virtual store tool405 provides feedback 2116 to cart review assistant 2102, indicatingthat virtual shopping cart 420 matches algorithmic shopping cart 1420.In certain embodiments in which machine learning algorithm 2112 is areinforcement learning agent, cart review assistant 2102 then uses thisfeedback to provide a penalty to reinforcement learning agent 2112.Method 2300 then proceeds to step 2326.

If, in step 2306 cart review assistant 2102 determines that machinelearning algorithm 2112 determined that algorithmic shopping cart 1420is not a good candidate for review, in step 2316 cart review assistant2102 uses algorithmic shopping cart 1420 to charge an account belongingto customer 105 for the selections he/she made during the shoppingsession and issue a corresponding receipt 1465 to customer 105. In step2318 cart review assistant 2102 receives feedback 2208 from customer105. In certain embodiments, feedback 2208 is a refund request submittedby customer 105 to virtual store tool 405. Virtual store tool 405 thenprocesses the refund request by virtually emulating the shopping sessionassociated with the refund request. In step 2320 virtual store tool 405determines whether the virtual cart 420 resulting from the virtualemulation of the shopping session associated with the refund requestmatches the algorithmic shopping cart 1420 that was used to chargecustomer 105. If, in step 2320 virtual store tool 405 determines thatvirtual shopping cart 420 does not match algorithmic shopping cart 1420,in step 2322 virtual store tool 405 provides feedback 2116 to cartreview assistant 2102, indicating that virtual shopping cart 420 doesnot match algorithmic shopping cart 1420. In certain embodiments inwhich machine learning algorithm 2112 is a reinforcement learning agent2112, cart review assistant 2102 then uses this feedback to provide apenalty to reinforcement learning agent 2112. Method 2300 then proceedsto step 2326. If, in step 2320 virtual store tool 405 determines thatvirtual shopping cart 420 matches algorithmic shopping cart 1420, instep 2324 virtual store tool 405 provides feedback 2116 to cart reviewassistant 2102, indicating that virtual shopping cart 420 matchesalgorithmic shopping cart 1420. In certain embodiments in which machinelearning algorithm 2112 is a reinforcement learning agent, cart reviewassistant 2102 then uses this feedback to provide a reward toreinforcement learning agent 2112. Method 2300 then proceeds to step2326. In step 2326 cart review assistant 2102 uses the punishment andreward values to refine machine learning algorithm, 2112.

In certain embodiments, feedback 2208 received by cart review assistant2102 at step 2318 includes the absence of a refund request submitted bycustomer 105 to virtual store tool 405. For example, in certainembodiments, if virtual store tool 405 does not receive a refund requestfrom customer 105 within a set period of time, cart review assistant2102 may assume that customer 105 determined that algorithmic shoppingcart 1402 was accurate. Accordingly, in certain embodiments, at step2320 cart review assistant 2102 may assume that if virtual store tool405 were to virtually emulate the corresponding shopping session, theresulting virtual shopping cart 420 would match algorithmic shoppingcart 1420. In some embodiments, even though virtual store tool 405 didnot receive a refund request from customer 105 within a set period oftime, cart review assistant 2102 may nevertheless randomly decide toinstruct virtual store tool 405 to virtually emulate the shoppingsession of customer 105, and to compare the resulting virtual shoppingcart 420 to algorithmic shopping cart 1420 at step 2320. This may bedesirable in certain embodiments where errors in algorithmic shoppingcart 1420 may result in customer 105 being undercharged.

Modifications, additions, or omissions may be made to method 2300depicted in FIG. 23. Method 2300 may include more, fewer, or othersteps. For example, steps may be performed in parallel or in anysuitable order. While discussed as cart review assistant 2102 (orcomponents thereof) and virtual store tool 405 (or components thereof)performing the steps, any suitable component of system 2100, such asexternal system 485 for example, may perform one or more steps of themethod.

While the preceding examples and explanations are described with respectto particular use cases within a retail environment, one of ordinaryskill in the art would readily appreciate that the previously describedconfigurations and techniques may also be applied to other applicationsand environments. Examples of other applications and environmentsinclude, but are not limited to, security applications, surveillanceapplications, object tracking applications, people trackingapplications, occupancy detection applications, logistics applications,warehouse management applications, operations research applications,product loading applications, retail applications, roboticsapplications, computer vision applications, manufacturing applications,safety applications, quality control applications, food distributingapplications, retail product tracking applications, mappingapplications, simultaneous localization and mapping (SLAM) applications,3D scanning applications, autonomous vehicle applications, virtualreality applications, augmented reality applications, or any othersuitable type of application.

Although the present disclosure includes several embodiments, a myriadof changes, variations, alterations, transformations, and modificationsmay be suggested to one skilled in the art, and it is intended that thepresent disclosure encompass such changes, variations, alterations,transformations, and modifications as falling within the scope of theappended claims.

What is claimed is:
 1. An apparatus comprising: a display; and a hardware processor communicatively coupled to the display, the hardware processor configured to: display, in a first region of the display, a virtual shopping cart; display, in a second region of the display, a virtual layout configured to emulate a physical layout of a physical store, the virtual layout comprising: a first virtual rack associated with a first virtual item, the first virtual item comprising a graphical representation of a first physical item; and a second virtual rack associated with a second virtual item, the second virtual item comprising a graphical representation of a second physical item; receive information indicating that an algorithm determined that the first physical item was selected by a person during a shopping session in the physical store, wherein the algorithm determined that the first physical item was selected by the person based on a set of inputs received from sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the first item was selected: display, in a third region of the display, the first virtual item; display, in a fourth region of the display, a first rack video captured during the shopping session of the person in the physical store, the first rack video captured by a first rack camera of a set of rack cameras located in the physical store, the first rack camera directed at a first physical rack of a set of physical racks located in the physical store, the first physical rack comprising the first physical item; in response to displaying the first rack video, receive information identifying the second virtual rack, wherein the first rack video does not depict that the person selected the first physical item; and in response to receiving the information identifying the second virtual rack: display, in the third region of the display the second virtual item; display, in the fourth region of the display, a second rack video captured during the shopping session of the person in the physical store, the second rack video captured by a second rack camera of the set of rack cameras located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks, the second physical rack comprising the second physical item; in response to displaying the second rack video, receive information identifying the second virtual item, wherein the second rack video depicts that the person selected the second physical item while interacting with the second physical rack; and in response to receiving the information identifying the second virtual item, store the second virtual item in the virtual shopping cart.
 2. The apparatus of claim 1, wherein the hardware processor is further configured to: receive information indicating that the algorithm determined that a third physical item was selected by the person during the shopping session in the physical store, wherein the algorithm determined that the third physical item was selected by the person based on the set of inputs received from the sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the third item was selected: display, in the second region of the display, a third virtual item, the third virtual item comprising a graphical representation of the third physical item; display, in the third region of the display, a third rack video captured during the shopping session of the person in the physical store, the third rack video captured by a third rack camera of the set of rack cameras located in the physical store, the third rack camera directed at a third physical rack of set of physical racks located in the physical store, the third physical rack comprising the third physical item; in response to displaying the third rack video, receive information identifying the third virtual item, wherein the third rack video depicts that the person selected the third physical item while interacting with the third physical rack; and in response to receiving the information identifying the third virtual item, store the third virtual item in the virtual shopping cart.
 3. The apparatus of claim 1, wherein the hardware processor is further configured to: receive information indicating that the algorithm determined that a third physical item was selected by the person during the shopping session in the physical store, wherein the algorithm determined that the third physical item was selected by the person based on the set of inputs received from the sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the third physical item was selected: display, in the second region of the display, a third virtual item, the third virtual item comprising a graphical representation of the third physical item; display, in the third region of the display, a third rack video captured during the shopping session of the person in the physical store, the rack video captured by a third rack camera of the set of rack cameras located in the physical store, the third rack camera directed at a third physical rack of a set of physical racks located in the physical store, the third physical rack comprising the third physical item; in response to displaying the third rack video, receive information indicating that the person did not select the third physical item during the shopping session, wherein the third rack video does not depict that the person selected the third physical item while interacting with the third physical rack; and in response to receiving the information indicating that the person did not select the third physical item, remove the third virtual item from the second region of the display.
 4. The apparatus of claim 3, wherein, in response to receiving the information indicating that the algorithm determined that the third physical item was selected, the hardware processor is further configured to: display, in a fourth region of the display, a virtual rack configured to emulate the third physical rack, wherein the virtual rack comprises a first virtual shelf, the first virtual shelf comprising the third virtual item and a fourth virtual item, wherein the fourth virtual item comprises a graphical representation of a fourth physical item; receive information identifying the fourth virtual item, wherein the third rack video depicts that the person selected the fourth physical item while interacting with the third physical rack; and in response to receiving the information identifying the fourth virtual item, store the fourth virtual item in the virtual shopping cart.
 5. The apparatus of claim 3, wherein the hardware processor is further configured, in response to receiving the information indicating that the person did not select the third physical item, to: attach metadata to the set of inputs received from the sensors located within the physical store, the metadata indicating that the person did not select the third physical item; and use the set of inputs to train the algorithm.
 6. The apparatus of claim 1, wherein the first rack video captures a portion of the shopping session.
 7. The apparatus of claim 1, wherein the information identifying the first virtual item comprises information associated with dragging and dropping the first virtual item from the second region of the display to the virtual cart.
 8. A method comprising: displaying, in a first region of the display, a virtual shopping cart; displaying, in a second region of the display, a virtual layout configured to emulate a physical layout of a physical store, the virtual layout comprising: a first virtual rack associated with a first virtual item, the first virtual item comprising a graphical representation of a first physical item; and a second virtual rack associated with a second virtual item, the second virtual item comprising a graphical representation of a second physical item; receiving information indicating that an algorithm determined that the first physical item was selected by a person during a shopping session in the physical store, wherein the algorithm determined that the first physical item was selected by the person based on a set of inputs received from sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the first item was selected: displaying, in a third region of the display, the first virtual item; displaying, in a fourth region of the display, a first rack video captured during the shopping session of the person in the physical store, the first rack video captured by a first rack camera of a set of rack cameras located in the physical store, the first rack camera directed at a first physical rack of a set of physical racks located in the physical store, the first physical rack comprising the first physical item; in response to displaying the first rack video, receiving information identifying the second virtual rack, wherein the first rack video does not depict that the person selected the first physical item; and in response to receiving the information identifying the second virtual rack: displaying, in the third region of the display the second virtual item; displaying, in the fourth region of the display, a second rack video captured during the shopping session of the person in the physical store, the second rack video captured by a second rack camera of the set of rack cameras located in the physical store, the second rack camera directed at a second physical rack of the set of physical racks, the second physical rack comprising the second physical item; in response to displaying the second rack video, receiving information identifying the second virtual item, wherein the second rack video depicts that the person selected the second physical item while interacting with the second physical rack; and in response to receiving the information identifying the second virtual item, storing the second virtual item in the virtual shopping cart.
 9. The method of claim 8, further comprising: receiving information indicating that the algorithm determined that a third physical item was selected by the person during the shopping session in the physical store, wherein the algorithm determined that the third physical item was selected by the person based on the set of inputs received from the sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the third item was selected: displaying, in the second region of the display, a third virtual item, the third virtual item comprising a graphical representation of the third physical item; displaying, in the third region of the display, a third rack video captured during the shopping session of the person in the physical store, the third rack video captured by a third rack camera of the set of rack cameras located in the physical store, the third rack camera directed at a third physical rack of set of physical racks located in the physical store, the third physical rack comprising the third physical item; in response to displaying the third rack video, receiving information identifying the third virtual item, wherein the third rack video depicts that the person selected the third physical item while interacting with the third physical rack; and in response to receiving the information identifying the third virtual item, storing the third virtual item in the virtual shopping cart.
 10. The method of claim 8, further comprising: receiving information indicating that the algorithm determined that a third physical item was selected by the person during the shopping session in the physical store, wherein the algorithm determined that the third physical item was selected by the person based on the set of inputs received from the sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the third physical item was selected: displaying, in the second region of the display, a third virtual item, the third virtual item comprising a graphical representation of the third physical item; displaying, in the third region of the display, a third rack video captured during the shopping session of the person in the physical store, the rack video captured by a third rack camera of the set of rack cameras located in the physical store, the third rack camera directed at a third physical rack of a set of physical racks located in the physical store, the third physical rack comprising the third physical item; in response to displaying the third rack video, receiving information indicating that the person did not select the third physical item during the shopping session, wherein the third rack video does not depict that the person selected the third physical item while interacting with the third physical rack; and in response to receiving the information indicating that the person did not select the third physical item, removing the third virtual item from the second region of the display.
 11. The method of claim 10, wherein, in response to receiving the information indicating that the algorithm determined that the third physical item was selected, the method further comprises: displaying, in a fourth region of the display, a virtual rack configured to emulate the third physical rack, wherein the virtual rack comprises a first virtual shelf, the first virtual shelf comprising the third virtual item and a fourth virtual item, wherein the fourth virtual item comprises a graphical representation of a fourth physical item; receiving information identifying the fourth virtual item, wherein the third rack video depicts that the person selected the fourth physical item while interacting with the third physical rack; and in response to receiving the information identifying the fourth virtual item, storing the fourth virtual item in the virtual shopping cart.
 12. The method of claim 10, further comprising, in response to receiving the information indicating that the person did not select the third physical item: attaching metadata to the set of inputs received from the sensors located within the physical store, the metadata indicating that the person did not select the third physical item; and using the set of inputs to train the algorithm.
 13. The method of claim 8, wherein the first rack video captures a portion of the shopping session.
 14. The method of claim 8, wherein the information identifying the first virtual item comprises information associated with dragging and dropping the first virtual item from the second region of the display to the virtual cart.
 15. An apparatus comprising: a display; and a hardware processor communicatively coupled to the display, the hardware processor configured to: display, in a first region of the display, a virtual shopping cart; display, in a second region of the display, a virtual layout configured to emulate a physical layout of a physical store, the virtual layout comprising: a first virtual rack associated with a first virtual item, the first virtual item comprising a graphical representation of a first physical item; and a second virtual rack associated with a second virtual item, the second virtual item comprising a graphical representation of a second physical item; receive information indicating that an algorithm determined that the first physical item was selected by a person during a shopping session in the physical store, wherein the algorithm determined that the first physical item was selected by the person based on a set of inputs received from sensors located within the physical store; in response to receiving the information indicating that the algorithm determined that the first item was selected: display, in a second region of the display, the first virtual item; display, in a third region of the display, a first rack video captured during the shopping session of the person in the physical store, the first rack video captured by a first rack camera of a set of rack cameras located in the physical store, the first rack camera directed at a first physical rack of a set of physical racks located in the physical store, the first physical rack comprising the first physical item; in response to displaying the first rack video, receive information indicating that the person did not select the first physical item during the shopping session, wherein the first rack video does not depict that the person selected the first physical item while interacting with the first physical rack; and in response to receiving the information indicating that the person did not select the first physical item: remove the first virtual item from the second region of the display; receive information identifying the second virtual rack; and in response to receiving the information identifying the second virtual rack:  display, in the third region of the display, the second virtual item;  receive information identifying the second virtual item; and  in response to receiving the information identifying the second virtual item, store the second virtual item in the virtual shopping cart.
 16. The apparatus of claim 15, wherein the first rack video depicts that the person interacted with the second physical rack.
 17. The apparatus of claim 15, wherein the hardware processor is further configured to: display an image of the person; and in response to displaying the first rack video, receive an indication that the first rack video does not depict the person.
 18. The apparatus of claim 17, wherein the indication that the first rack video does not depict the person comprises an indication that the first rack video depicts that a second person selected the first physical item while interacting with the first physical rack.
 19. The apparatus of claim 15, wherein the first rack video captures a portion of the shopping session.
 20. The apparatus of claim 15, wherein the hardware processor is further configured, in response to receiving the information indicating that the person did not select the first physical item, to: attach metadata to the set of inputs received from the sensors located within the physical store, the metadata indicating that that the person did not select the first physical item; and use the set of inputs to train the algorithm. 